No. It depends on the situation, what was said, who said it, who heard it, etc. Even if it doesn't affect you emotionally, their remarks can damage your reputation, which can hurt you in the long run.