You're in a pickle of a situation, Jb....
How do the respective spouses, both yours and the husband of the exgirlfriend feel about the marriages?
I feel I can respond to this because I stayed unhappily in a loveless marriage for 20 years, mainly for my daughter. My ex knew I was unhappy but refused to discuss divorce, instead threatening to take our daughter from me. What would happen to the children if you two divorced in order to be together? If you could provide the happy loving home of which you speak, that would be wonderful, wouldn't it!, but bitterness ensues with such divorces, also affecting the children.
I hope things work out for you.
Patty