Gypsy....
I can relate to your situation. Before my divorce I was having an affair with a woman, who was also married. All I can tell you is please be careful, with children involved there are too many lives at risk. Please take a moment to consider the impact of your actions on them.
I can also relate to the problems/issues that the man you are seeing has with sex and sexuality. For me I became much more interested in sex because I spent years in a marriage that was devoid of real passion and sex was far too infrequent. The problem is that sex became an escape for me as I endured the pain of the loss of my marriage and the resultant depression. Sex was my way to self-medicate. I entered in to relationships for teh wrong reasons. And, I lost a wonderful relationship because of my sexual exploitations. It wasn't until I hit rock bottom that I got the help I needed and replaced the bad "esacpe" behaviors with healthy and productive behaviors. So, if you don't d anything else make sure that he gets help. He may very well be depressed and sure shows some signs of a possible sexual addiction.
You are involved in a very difficult situation. First, I think that it has a high probability for disaster... for him, for you, for his kids and his wife. Second, he is in a period that is very chaotic, he needs time to figure out what he wants personally. You may be a way for him to cope. But, it may or may not mean that he is willing, or able, to be your life partner. Affair relationships are not like real, committed relationships. You don't have to deal with the day to day struggles of married life. You tend to be apart from each other for longer periods so when you do come together both parties are typically happy to see each other. Third, you said he wouldn't leave because of his kids. That makes me very nervous. He wants his cake and eat it too. You just can't do that. He needs to make a decision one way or the other. This is not healthy for his kids, nor any one else involved.
I can't tell you what to do, but for the sake of his children he needs to make a decision one way or the other. Either committ 100% to his marriage or end it. And, if he did decide to end his marriage, I would suggest taking a good LONG time before taking your relationship to the next level. Trust me, from experience these "rebound" relationships often end in disaster.