I don't think this is the case at all. People are afraid of him. He simply has clout and like any bully people are afraid of the repercussions should they offend him or attempt to push back.

He may also be a narcissist. they tend to have a great deal of charisma and influence. They have a way of gathering people around them - usually preying upon their weaknesses to do so. They live for superiority. Everything is about them. By refusing to move, he hi-jacked your misfortune to make it about him.

Do you have a Manager? Do you have an HR department? Do you have an EAP?

Each of these are obligated to do something about this. Start documenting these incidents and present them to your superiors. If your workplace refuses to do something about this then I suggest you go the 'Human Rights' route.