Just a thought... Maybe you both need some Therapy , On your own, not as a couple at this point.. he has baggage and issues to work through and you do also.

I hope you take time to care for yourself, It's not all "about him"