I think it's time to ignore what coworkers have to say and address this with your direct supervisor.

The next time you have a review or a sit-down with them about your performance, I would address is tactfully by asking "I would like your feedback about my ability to be a team player. What suggestions do you have for me to be more approachable?" If your supervisor thinks you're doing fine, take it as that. If your supervisor has feedback, process it and figure out how to incorporate it into your working relationships.

I think that coworker was trying to start gossip. Gossip is not something you want to take part in, even if it was about you.