Sadly there isn’t much one can do but think it’s not about you really

But again it depends. Are these people family members? Friends? I’d look for new friends. Family you can’t choose so I guess I’d keep my interaction with them to a minimum. My husband has some relatives like this. We keep interaction to an absolute minimum.