Yes, you should be worried - if a man will hit his mother he will not think twice about hitting another female (wife & child included).

It has always been said... If you want to see how your man will treat you watch how he treats his mother.