I'm glad to know your age, Double!
23 is very young and you have your whole life ahead of you. So many things to look forward to!
A man who tells you he is unsatisfied sexually in his marriage, who exhibits interest in you....is trouble! As someone else here has said, he may even by lying! Even if what he says is true, it is morally wrong to become involved with such a man!
You have related a troubling history here, and it sounds like you're vulnerable.
Your best course of action would be to abstain from becoming sexually involved right now.
Patty