I struggle with this too, although it admittedly has nothing to do with my bipolar.
I have BPD as well, and it stems from that.

The worst my BP does is me isolating while depressed, but by now everyone knows to give me some room to breath until I indicate otherwise.