Distancing yourself from these people sounds like a great idea. I would also work on sticking up for myself. I am not sure if that is the right phrasing, but if someone says something rude to you, or something you don't like, you can tell them and put a stop to it right then. Cultivate a sense of outrage at this shocking bad behavior -- because it really is shocking for adults to be telling other adults that they are ugly or stupid!

A better question might be "how can I make people stop saying these rude things to me" instead of "how can I live with the horrible things people say to me?"

"Wow, that's a really hurtful thing to say. Why would you say that to someone?"
"Please don't say things like that to me. How would you feel if I said that to you (or to your daughter/grandchild/whatever)?"
"That hurts my feelings."

I wonder if fake-it-til-you-make-it works for this situation? When we don't have enough self-esteem in the first place, we tend to put up with rude things people say to us, which then serves to confirm our low-self esteem. It's a vicious cycle. If we break one part of the cycle, the part where we let people say rude things, will that help improve our self esteem? I wonder...