I have AvPD - I totally understand this! It's fairly common for people with AvPD to be really insecure if they have friends with lots of other friends, or even friends with just a few other friends. You always feel like you're going to be bumped out of the circle.

I wish I had great advice for you. The only thing I can say is do stuff. Be interested in things, and you will be interesting to your friend. Talk to other people with your same interests, and perhaps you will find someone who can be another friend for you.

Also - you might not be his "favorite friend" but you are a good friend. That's pretty special! He wants to be with you, so focus on that if you can.