Wow. This post reminds me so much of myself, and I've had a therapist say the same thing in the past and can definitely understand why this would trigger you.
If you feel comfortable enough telling your therapist that the comment hurt you, then maybe it could strengthen the therapeutic relationship. Hopefully your T will try to understand more of where you are coming from and validate you moving forward. Based on personal experience and hearing from others, I understand why having Bipolar could interfere with social interactions, especially combined with being let down (and/or traumatized) by others. It's difficult to break the cycle. It is not your fault, and it is entirely up to you as far as how much or how little you choose to socialize. There is no right or wrong. Readiness is often a factor too, especially since you've gone through some negative experiences with others in the past. But there is nothing wrong with being an introvert. You are not alone.