It might be because of the clothes - that's not a critism, just trying to think of other reasons , well it can't all be that because of that, but maybe you don't fit into a clique.

So people might think they won't have much in common - sometimes people are shallow.