That is tricky. My thought is that this may only have to do with Tom, because why did they not ask you the same? I wonder if someone in management wants to build a case that Tom can't do the job to get him to retire early or let him go. Could that be possible? I was in a job where my manager purposely gave my coworker a difficult project to make a case to HR that he didn't have the skills for the job (this employee didn't have great skills). He ended up using that to get him transferred to a lower level position.

If I was in the situation, I might talk to the manager about what you heard about Tom. I think it will continue to bother you until you know what is really going on.