Your husband makes me want to revert away from my pacifism for a day. This sounds like my upbringing with my parents; this sort of thing always happened and it got me all twisted up with my mom. No parent has a right to do that to their children and no husband should ever do that to his own wife. If you're trying your hardest, you're enough. Plain and simple. Don't let this guy get inside your head like that, by using your kids as his weapon.
You're human. Human's make mistakes, even parents. I'm pretty sure we would notice here and I'm sure it would break your kids' hearts. Even with my terrible relationship with my dad (that he caused, which isn't the case for you), I would mourn and I know I would probably cry.