My 2 Cents.

Sounds like he doesn't like working there and and doesn't get along with anyone but you.

I think he may have said it innocently that you're the only one he likes to work with.

I've been at work places where people are poisonous and there would be one decent person in the whole group. I would tend to work closer to that one person because they were somewhat of a safety blanket away from the poisonous workers.

If it is something more than that, just make sure he's not married or has a GF.