Maybe the critical gossips are the problem, not the one they are talking about. If they are unpleasant to you is there someone else that you enjoy being with you could do things with?