If I truly wanted to end the friendship with him only, and not her, I'd flatly and without emotion say, "I like your mom, she's cool. I want to be her friend. You, on the other hand, I'm over."
He'll likely be a bigger arse, but you do have the option of calling the police if he's still being abusive or threatening