I go to a peer support group, like you I don't know if it has helped or not. For some reason I feel somewhat responsible to be there to try to help others. But in your case if it is causing you anxiety then it is probally not the best, I might not be getting much out of my group but there is no down side that I see either.

You don't have to have some confrontation with the group, you can just stop showing up, though I would suggest sending at least one member of the group either the therapist or just one group memebers and let them know you are OK but won't be attanding meetings any more. You don't have to go into details, you don't have to justify yourself to others.

You just need to do what is best for you.