My feelings about this echo some already expressed.. I used to care what people said about me but don't anymore; it's been a progressive effect for me. For many years I would
say I didn't care because it was what I wanted to manifest, but now I just genuinely don't. I mean, I do care what people think of me BUT, I expect them to judge me on me only, and my actions, and if they are not wise enough to know to base opinions on more well reasoned assessments, then the best thing I can do to help them join the world of kind and intelligent discourse is not honor the b.s. they are generating. Beyond that I can't worry about it.
That said, there are certain actions, which could perhaps be called inactions, that I've taken to protect my own image -- in the sense that my life is not 100% an open book. However warm I feel towards a person, I don't want to burden friendships with too many personal truths that I would rather not have widely distributed, because their trustworthiness does not preclude the possibility that they could say something in error based on what I've shared, so why burden them with rope that can hang us? People love to talk, so I'd just as soon their fires not be fueled by my own most sensitive moments in life.

But, this is not hard work for me, as I've gotten quite good at it.
And I'm with Iris & Trippin, where discussing the private affairs of others is concerned; I think not taking part goes a long way towards having peace in this area. For the most part people don't talk trash with me, because they know I don't do it, and I have to think my standing for that lessens the possibility of others talking about me in the same way, since there's no potential tit for tat involved.
*edited to add:
This dude actually has a good way of saying the same thing, and quite a bit more concisely than me:
"My philosophy is: It’s none of my business what people say of me and think of me. I am what I am and I do what I do. I expect nothing and accept everything. And it makes life so much easier." ~Anthony Hopkins