He shouldn’t have compared you with anybody to begin with.
I can guess the post as a boss is not easy (I wouldn’t ever be able to play this role) but I do think a worker deserves respect and being listened. They are the most valuable element in a place work.

You can’t guess how many times my partner has complained his boss about the conditions and treatment this “element” has with his coworkers. He (the boss) doesn’t take actions against my partner because he knows that he needs him.

The key is that before this scenario there are people like my partner who takes these discussions without giving a lot of importance and then, there are people like you or me who wonder and question about everything. Even blaming ourselves for daring to “rise” our voice.
I understand your concern.
I met a man from the UK, working in Germany and he was a model as a boss. He took care of all his coworkers. He talked to them when they had a problem and listened to them. This is a real leader. But, of course, he has a great personality, bubbly, encouraging, understanding and few boss are like that.