Scott;

Are you slightly built or short in stature? I'm trying to get a mental image of you so I can see where your vunerability is being projected. Can you look a man in his eyes as you walk by? Do you avert your eyes and walk with your arms close to your body? Hands in pockets? Head down?

The way you project yourself is important. Men have a way of knowing that they can imtimidate another.
Where you ridiculed by older men when you were young?
All these things can affect you without you even being aware if them.