Disclaimer: I am not diagnosed with Avoidant or any personality disorder. Just bipolar 2.

I do, however, relate extremely closely to AvPD and feel that it fits my personality quite perfectly. I feel frustrated by it, because it outlines all the things I struggle with in a very concise way. I don't actually have social anxiety, as I like people and I do like to be around people. I just can't DO things around people and I don't trust them and worry about rejection way too much. My T decided quite early to not bother working on managing my bipolar symptoms as I've got good control there, but has instead decided to work on my low self-esteem. We're doing DBT. He said CBT, but he's working with a DBT book so I think he made a mistake somewhere in there

I wouldn't be surprised if you were to start working on your self-esteem and then work on some exposure therapy challenges. To me, that sounds like a logical progression.