Straight but gay acting

OK, this may seem weird but it's something that I think about a lot.

I'm straight but I have this obsession that people think I'm gay. It's not entirely unfounded because in the past people have asked me, I've been propositioned by men and one friendship ended when I realized that the guy probably had a crush on me (or just enjoyed making homophobic jokes at my expense).

If I was gay this would all be good news but I'm not. I'm not homophobic so it doesn't really bother me if some people think I'm gay. I've come to assume that I either look or act gay in some way. I'm soft spoken, I seem to always end up listening attentively to women (which I enjoy) - at work right now, there's a girl who I fancy like mad and who talks to me a lot but in the back of my mind I'm scared to make a move as feel she possibly just sees me as a 'safe' guy.

I've had girlfriends and full relationships but they never last for long and there are long lonely years between them.

Maybe a lot of it is insecurity and it would vanish if I was in a long term relationship. Just wondering if anybody else has had similar issues?