A lot of friends have drifted away for me also.
(But some of that would have happened anyway. I am in my forties now, and the crowd I used to hang out with when I was young... well, we have grown apart. Going in such different directions. Sorry, digression).
It is traumatic to feel judged and to experience people you consider friends drifting away. Sympathize completely with you there!
With me it is not so much mania, I become hypomanic, (BP II). But I have long periods of not being able to socialize at all, and I make and break appointments I told people I would attend. In the end, that has led a lot of people to turn away from me. The few friends I have left accept and understand how things are with me, and these friendships are truly meaningful to me. Treasure your true friends, because that is what they are, treasures, and truly good people. But it is heartbreaking to experience rejection.