I think given how he’s responded to you, it would be wise to drop him. He sounds possessive, at best. I’m not saying it’s definitely the case, but I wouldn’t be surprised if these bad experiences he’s had are because of how he is - his expectations and demands (and who knows what other behaviour you haven’t witnessed) scaring people off, but he might not be willing to admit that. Maybe it was even him who cheated - people who cheat often do accuse others of doing it to them.