I usually try to sort this out with I statements - so I say (very politely and apologetically, because I'm British, and I would generally rather die than make a fuss) "Excuse me, I'm sorry to bother you, but are you aware that I am feeling quite a lot of stress because of what you just did?". Leave it at that, make it non-acusatory, but get your point over. Then it's his ball. Don't fight.

It's hard, and it makes you feel vulnerable and horrid, but it does work better than my usual overreaction, black and white thinking, catastrophic scenarios and screaming approach to people.