It sounds to me as if someone's playing "Heads I win, tails you lose" with you. Family therapists figured out long ago that the "identified patient" is almost always just the scapegoat (thanks, lynn09) for other family members who are less aware of their problems and more successful at passing them off to someone else. That your pastor seems to have missed this, suggests that (a.) he has limited skill as a family therapist and (b.) he's doing neither you nor your husband any great favors.

Good luck, tryingtobeme. I suspect your best bet would be to find counseling somewhere where you're granted a level playing field.