You need to remember that this pain you are feeling right now is a pain that you have caused your husband to feel over and over again.

He was bound to crack at some point.

That's not to say that what he is considering is ideal or appropriate but if you want your marriage to survive you are both going to need to put in a lot of work if neither of you are willing than perhaps it is time to accept that it just can't work.