I don't know. Some ideas I have are that like white men from the 1950s were portrayed as in charge, authority figures. You know that whole "the man" persona. "The man" is trying to keep us down. You know? Also, in media they are often portrayed as rigid in their beliefs and thinking, especially more currently.
So, I don't know. Or maybe at some point you had some sort of run in with a man like that when you were a kid. Maybe you don't even remember it. But, maybe something happened that stuck to your subconcious.
Just ideas I had in thinking about it.