I don't want to make you feel bad and regardless of which party you are in a situation like this, it is painful and hard on all sides. However--I sound exactly like your "boyfriend's" wife. In fact, it sounds so close to my own experience that is is scary. Here is what I would ask you to consider:
My husband's story about our relationship to his mistress, was not what I thought was happening at all. When I heard how he had described out relationship, and the things he had told her about me it was so far off of how he interacted with me and the day to day reality of our lives together, it was the Twilight Zone. He fictionalized and made up many imagined details to make it seem like I had been terrible and had driven him to cheat on me. What she thought about me and our relationship was just not the truth. Is your boyfriend's version one that lets him justify what he wants, or is it reality? Is your version of your marriage the reality you describe, or may you be looking for negative details in order to justify your behavior?
You say you love him but to go from the sneaking around and kind of taboo excitement you might be experiencing to having to clean up after him, drive him around, help him with technology, deal with his family, financial issues, and just regular life stuff often is not that much better than the relationship you are wanting to leave. We are basically ourselves in every relationship. Be careful.
If you do get together, both of you have acted in ways that will forever affect each other's children and that may be a huge source of tension as you move forward. I have three adult children and two of them refuse to have anything to do with my husband.
If he cheated on his wife? And he lied to her for this long, how can you trust what he says to you?

I have found that, sometimes when we are feeling bad about our relationship, we think moving to a new one will make us happy. This is not generally true and after a few months together, my husband's mistress dumped him. He wanted to come home and acted like a victim. It was hard, but I didn't let him. Now he has lost the trust and closeness of his family and the woman he put above everyone else.

My advice is, get out of an unhappy marriage but spend enough time with yourself that you choose a new healthy romantic situation once you know what you need.