While I don't know what you mean by them "going home"... if they are his kids on that front I definitely would defend his right to have them around, but that's not where I see the problem in your situation. I see you building up resentment toward the kids because although he takes the responsibility to "have them" around, he doesn't seem to do much beyond that as their father and even more so, your partner. The problem seems to be stemming from him passing on the responsibilities that should not be yours in the first place. You're not even technically their step parent, so that should not be soemthing you need to do, let alone take care of his lazy ***.

I'm sorry but you need to step it up with him and tell him to get a job, take care of his kids or get out. seriouslly I don't see him doing anything differently until you give him the ultimatum.