Yeah, I wouldn't let it bother you. Men who are good-looking, dress well, and hang out mostly with women are often assumed to be gay. But I bet it would be a pleasant surprise to some women to find out you're straight.