How are they supposed to feel good about themselves if they can't be down on you (or somebody) for doing something wrong?

I'd say: recognize that it's they who have the problem, that they probably aren't inviting you to help them let go of it, that you don't need their good opinion, and that one of the best things available to do is just steer clear of them.

I suppose you could, if you were so inclined, give them a parting zinger like, "But it's fun!" If I did that, though, I probably wouldn't stick around long.