I believe that when you're truly comfortable with yourself, other's opinions don't matter.
So I wouldn't let other's opinions have so much affect on you.
I've walked down the street (I'm a female) holding a girlfriend's hand, without a care in the world..
I've also (At 22) walked down the street holding a 39 year old man's hand, happiest I've ever been.
I didn't care what anyone thought at all 'cos I was happy in my own little world.