I wish that I knew the answer, because I'm in a similar state of mind.

I go through times that aren't too bad, but at other times, it seems that being around
more than 2 or 3 people causes me to get increasingly agitated. The
more people, the greater the level of my agitation. It's definitely not anxiety or
fear (in my case), but simply "Get the heck out of my personal space!" If I know
the people, like co-workers, it isn't quite so bad, but sometimes I just can't
tolerate strangers.

I miss out on a lot of social functions because of this.

When it gets too bad, I have a nearby park that I go to in order to get away. It
isn't at all crowded, and I find that spending some time there and focusing on
nature stuff (I birdwatch) tends to "reset" my mood to some extent.

The loneliness, I still have to find a way to deal with in the real world, but getting
away from everything for a time tends to alleviate my discomfort around groups
of other people.

I know that's probably not a lot of help, but maybe it's a little comforting knowing
that you're not alone in feeling the way that you do.