Thank you, thank you, thank you to all. I really appreciate the different perspectives and input.
Leed - to answer your question, I don't think I am constantly on his back. I do say things when it is a safety issue (e.g. please don't stack papers on the stove) or something to do with my son. That was what happened last night. He promised our son he would play with him and I heard my son ask at least 5 times for him to come play with no response (he was sitting on the couch playing a game on his phone) and I said "do you hear him asking you?" which then led to what I described in the original post.
Lido - Thanks for your perspective; I appreciate hearing from someone who grew up in a similar situation. And, yes, I do have family that could help.
I guess I still need to think about everything some more. Last night I got to thinking about why I am so scared to end this relationship and I kept going back to that phrase about the devil you know being better than the devil you don't. Thanks, Iluvcats, for letting me know that it is possible to start over with small kids and it can get better.