I think you have the right idea. If saying something to them (calling them out on it) isn't advisable...then you could either let them know you're uncomfortable with that sort of belittling talk, or................just walk away.

There is another phrase you should be aware of, besides the one about people talking about others behind their back -- and I agree with you on that one--and this phrase should make YOU stop and think.

Birds of a feather flock together.

If you don't want to be labeled equally judgemental.....get the flock out of there, when the talking starts....or before, preferrably.