It does sound unsupportive.

Only thing I can think to suggest is maybe making some other arrangements for outings with the express notice it is 'open to all', people may then have an opportunity to socialise.

Sometimes sadly it is good self-care to walk away from certain situations however.