I began writing my reply before your post, red panda, but you are definitely right about it not being directly related to avoidance. But it correlates with a number of different mental issues (50% of chronic substance abusers are said to be alexithymic according to studies and over 50% of people who are anorexic/bulimic).

In this case I think the lack of empathy thing doesn't necessarily mean you are in the sociopath direction. I think in this case it has to do more with not being able to accurately and intuitively appreciate someone else's emotional point of view ("impaired theory of mind"). Similar to the "lack of empathy" that is said to be a significant part of asperger's syndrome or HFA. My understanding is that people with aspergers tend to be significantly less manipulative than average, but then they might incidentally say or do something insensitive that is hurtful because they have a hard time seeing the emotional point of view of others. A study that wikipedia references found 85% of people with ASD/Asperger's were "impaired" (alexithymically) and 50% were "severly impaired" so there must be a differentiation between two different types of "lack of empathy" (evil vs. well-intentioned).

Sorry for the lecture-sounding post, but my point is I think it's more like a lack of intuitive understanding of the emotional intricacies of other people than a lack of caring who you hurt or manipulate as in AsPD, NPD or sociopathy.

Maybe we AvPDers all have our own reasons for being uncomfortable with other people in general or only of non-superficial relationships and this is just one possibility. AvPD seems to be a collection of symptoms with no specified root cause.