I do feel more drawn to people that are shy and introverted in hopes that they are able to understand me. Loners make me think that I would be more important to them and not just a third wheel of a friend.
They don't all have to be boring, desinterested or different from yourself! Call me naive but I do believe there exist people that you can connect with on a deeper level. It's just really hard to find and AvPD won't help with that.
I get your one on one situation and often critically overanalyze myself if I'm saying anything wrong to the point of mental breakdowns. If you truly trust someone it helps to give them the benefit of the doubt that they're busy if they aren't replying etc.
If you realize you put in more energy than you get back, you need to adjust your own expectations to reality and tone back. It has nothing to do with showing your avoidance or not being good enough. 'Normal' people also experience this but they chose a more healthy way of dealing with it.