Are you seeing a therapist or have access to one. My suggestion would be to make an appointment and tell him you would like him to come with you. Then you can talk about how you are feeling with a mediator present. Also, it may encourage him to seek help when he realizes how much this is effecting you.