Wow, I think I agree more and more with Josie. You are not making this better but worse. Why would he want to go to you and not mend his broken marriage instead of destroy it?
You are not presenting the attractive package of fun he signed up for, his reason for seeing you by the way. You are supposed to be a distraction from his reality, to add a little boost to his ego and not someone to listen to cry about how much you suffer because of him. He is right to ask you to go talk to someone else. You should.

Don't wait around for a married man to divorce. That is wasted time. If he divorces while you are out living your own life, then he can come find you if he is still interested. Then he will be yours and not until...

IMHO you need to get a life, stop hanging onto the scraps of his life that he throws your way.
This whole situation is not fair to anyone. Get earplugs and stop listening to what he does in his the life he built with someone else. It doesn't have anything to do with you anyway. He is living, you are self destructing.