If your wife/kids are happy and your wife knows of your limitations and possibilities, then it not anyone's business. They shouldn't tell you to leave for her sake especially if she/kids would be unhappy without your presence.

I'm wondering - isn't anger an emotion - so do you feel angry? If your wife/kids make you upset - can you control your actions? When a psychopath does something evil - are they feeling mad or angry at that moment? I know your father was mean - maybe you're doubting yourself because of those negative internal tapes from your past. Just take it day to day - keep the communication with your wife open. If she loves you and is happy - let this be the barometer of your relationship.