I am not a great one to give advice because I have never been as assertive in my marriage as I should have been but you might want to ask yourself this...
What do you think will be best for your children in the long run? I am a mother of two and even though my children are grown, the most important thing in my life is that they are healthy, successful and can follow their dreams. I don't think this feeling every goes away. My marriage is important to me but knowing what I know now, I would give it up if I thought it would make my children have better lives (since they are in their 20s, I think it is too late for this to make any significant difference in their lives). So I would try to unemotionally assess: Is he a good father? (Is kind and spends quality time with his children) And is he a good provider? (Sounds like he isn't) Since you have only been married 5 years, I recommend leaving sooner rather than later (if you think that is the best decision). The younger you and your kids are, the easier it might be to make the transition.