He sounds like he has a lot of baggage. You are not his ex-wife and it seems unfair to me that he is loading what is essentially his own issues onto you.

I have had a similar experience in the past, in that a partner projected a lot of his feelings about his ex onto me (not about sui, in our case he said I was clingy like his ex). Remember this is absolutely about his stuff, and don't take it too personally.

Good luck with your decision.