I have to agree with Sarahsweets. It sounds like he has anger issues of some kind and the kids are the target. They have little way of protecting themselves from this and are probably hoping you will do something about it, even if they’re not articulating it to you. If your husband is also only being rough when he can’t be called out for it, this also indicates he can choose how he behaves around them.
Doing right by the children would mean removing them from his presence.