(((((((((LK)))))))))
I'm so sorry you're being put through this, especially by someone who's supposed to be your friend. To be blunt, I wouldn't even "slowly downgrade" him to acquaintance. I would make it very clear that he was not to come near me or my family again. I agree with Elysium, you and your wife presenting a united front would be probably the best option, with the foreknowledge that there's a possibility he'll react with anger and you should do your best to keep your cool if he does. Talk it through with your wife first and make sure she's comfortable with telling him outright to back off, and come up with a plan together as to how you're going to approach him and what each of you wants to say. If you're calm and organized about it, it's more likely to show that united front than just plunging in next time he's over, or putting your wife on the spot.
The things he's said about his wife are terrible, and depending on your relationship with his wife, you might want to tell her about it if it worries you. If you don't feel comfortable telling her yourself, maybe you could ask your wife to speak to her woman-to-woman instead? Just a thought.
Good luck with all this!