I have the same problem of not being able to defend myself from the rude people; in my case it is more often an unkind remark from a normal person.

Contrary to putting the blame on him, I would think about why he that power over me and why I feel bad and why I could not defend myself.
So, when we get hurt, there are two things to look for:
1. Why I feel so bad when somebody says so and so to me
2. Why I cannot defend myself

For me, the answers to these questions were:

1. Why I feel so bad when somebody says so and so to me

The words that hurt us are what is true for us and we are ashamed of and trying to hide from other people. I will not feel bad if somebody calls me 'unfit, lazy person' because I know I am not. But I will feel bad if somebody calls me a bald, socially awkward person which I am Chances are that we are berating ourselves internally and hurting ourselves. So, we need to be more loving towards ourselves so that we do not feel our drawbacks are worth feeling ashamed about (like I being bald or socially awkward).

2. Why I cannot defend myself

In my case, I have been a pleasing person all my life and my fantasy was people are very powerful and if I please people, they will not hurt me. I have spent most of my social time in pleasing people rather than building the requisite skills of defending myself verbally or identifying why I am afraid of talking back to people.
For me, any conflict is an invitation to a group of people to humiliate me. So, that's why I have not built any backbone till now. So, I have been running away from conflicts and talking back will create one.

So, maybe you can find out answers to these questions for you and it will help you.