When I was a kid, I was wrongly labelled with the same thing. Except with me, it was simply quilt by association because my brother is autistic and because I was something of an outcast and not interested in traditional girlie things. The mistake haunted me for nearly a decade because I thought I was just that..."special."

Our society places so much value on a certain set of social skills (often that don't reflect a person's real personality) and outgoingness that telling a person they don't have these is really an insult, sadly. It is like shoved them into a special, you're not part of our world corner. This attitude pisses me off beyond belief,

You seem like an intelligent, well-articulated person. Psychology needs to stop telling us what is wrong and start helping us work with our strengths to reach our goals.

And your family sounds insensitive and like they're projecting their insecurities onto you, like a scapegoat. I am sorry.