Next time he says something like that, tell him that you prefer to only talk about work tasks. If he continues bothering you, talk to your boss and also document what he says and when