In my life, it really depends. Taken a lot of therapy, however, on my part, to reach a point where I can pinpoint what it is about what is being said, and realizing what I need to do about it, in the moment.

Are these irritating comments, coming from friends? family members? randomly placed? Sometimes who is saying something, can determine whether it's something you can just walk away from.

For instance, when I am in a large gather of people, at an event, etc. If I hear chatter that just tugs at that level of irritant, I can move away from it.
If it's a friend, I can say, hey! That's out of line. Same goes for family. Then I can say, listen, I didn't appreciate hearing that.