Yes, why you prefer your own company (in my case, because I felt the other people were ijits) can make a difference. Condescension (with or without rain :-) is not a nice personality trait. Hopefully 20 years of therapy got rid of a good little bit of mine.

Allegedly humans are "social" creatures so deliberately not doing stuff with others (being invited and always turning down or not being invited because you haven't made an effort to take an interest in others so they include you and then convincing yourself that, "I didn't want to go anyway") if it "sticks out" can be seen as others as anti-social or avoidant I guess.

I'd say I was pretty avoidant (sometimes still am). My stepmother was big on telling me, "no one's going to come beat down your door!" which I got very tired of hearing?