Jinny, "in love" feelings don't last a lifetime most of the time. You love him. Does he love you? Do you have mutual respect and honesty? If so, you have half the battle won. How about communication? We all need to work on that, it seems. Is he abusive to you or to the kids? If he's not, then I would say that the kids have a right to grow up with both their mom and dad in residence. They need and deserve both.
Where do you expect the trouble to come from? Is it a certainty, or is it something you just worry about? If you KNOW beyond a shadow of a doubt that there is trouble coming your way, then the two of you can prepare to meet it head on. Otherwise, don't borrow trouble.
Hope I've given you something to think about. There's more that I can say, but I'll leave it for a later time, if that's okay.
Please take care of YOU.
__________________
Psalm 119:105 Thy word is a lamp unto my feet, and a light unto my path.