obviously you have to talk to your husband if it's bothering you so much. if you don't think you can talk to him alone, maybe he can come to a counseling session with you - your own therapist if you have one or maybe find a couples' counselor. but you won't feel better unless he'll discuss it with you. otherwise you'll spend your time assuming what he meant when maybe all he meant was he was starting to get discouraged that you'd ever feel better - which is a reasonable response to the situation.

also, sometimes people just say things that they don't know will come off a certain way. and definitely don't let this one comment undo the progress you've made :-) you WILL get better, it just takes time and work.