I think I would focus on the 20% who don’t have a problem with you if that’s possible. I would limit my exposure to the others, is that possible? What are your husbands thoughts? It involves him too as they’re wives of his friends.
I honestly think life is too short to waste on people like this, my personal thoughts are that it’s better to have fewer but genuine friends, what you are describing here resembles nothing like friendship. No wonder you’re stressed, you deserve better people in your life.