It's simple. He is a hypocrite. He is so focused on finding fault with you now (and in turn, triggering your own neurotic "guilt") because he has focused so much on your past.

And you mentioned fantasies. What, he was fantasizing about you with other men? If so, that is the source of it, right there. He found out how many you were with and can't cope with his own guilty conscience.

Him wanting to be involved in all aspects of your life like that isn't enduring, it's a form of control.