I think it sounds like he has some jealousy issues that he needs to sort out on his own. I'm sorry to be so blunt, but for me this would be a dealbreaker. I couldn't stay with a man who was obsessed with my past.