You are bisexual though? So they are half correct?
I don't think people are dumb. Girls love gay guys and men that know how to dress nice.
Do you clearly flirt with women? Do you make it clear you are into them?
I once really liked a guy and he claimed to like me too but didn't ever make a move so I asked him if he was gay because his words and actions did not match. Turns out he was very very shy.. Or so he said..
There's also that thought that if a man is too good to be true he must be gay..
I wouldn't call people dumb over it.. That won't help you.. If you want women to respond to you differently you need to act differently..
(This is something I am learning myself.)
The common denominator is you.