Des...YouOme, Hun!
I have read all this thread, and you have gotten some excellent advice. I really can't add to it, except to say I somewhat identify with your experience. I did stay in a loveless marriage for 20 years for my daughter. In retrospect, I think I should have been more assertive to gain independence earlier on, but i felt I had no choices.
I never had a "physical affair," but I did emotionally attach to a man, who rejected me because I couldn't consummate the relationship. It took me a good 10 years, all the while still married, to recover from it. My husband knew of this, and with the nature of our marriage, it didn't really matter. I stayed.
I have now learned to be self-sufficient without a man. It's good for me , but I'm much older than you.
The man with whom you have had this affair sounds, quite frankly, like a loser, and more like an "escape" from your present situation.
If you do choose to leave, try to follow the others' advice here and do it for yourself, your independence, and establishing your own self-esteem and for your children.
Patty