Unfortunately, that's just how work is. People talk. I overhear coworkers talk about other coworkers, even managers talking about other managers. Not sure how to avoid this except to switch jobs, but people still talk.