He is miserable and senses that your life is going just fine, so he is trying to upset you in hopes you’d be just as miserable as him. Some miserable people (those miserable due to their own wrong doings) can’t stand a thought that others might be content and at peace let alone happy..

I recommend you file for divorce or at least legal separation so there would be child support order and he’d have to pay it. It’s not optional to support one’s children. Working also isn’t optional for the majority of us, regular people. You want to eat, you work. He knows you’ll take care of your kids while he’s doing who knows what. He has to be financially responsible for his minor child