This breakup is still so fresh and raw for you and its not allowing you to set boundaries. You and him need to agree to a set visitation schedule.. and not him coming to your home to see them.. He can come pick them up and go somewhere else. He gave up the right to call that place "home" when he left and cant just use it to make things "convenient for him." Its going to keep confusing you if he has free rein to come and go as he wants.

Dont fall into the rabbit hole of doubting his motives. Right now focus on taking care of yourself and your kids.