It does sound rude but it’s possible he meant he will not miss the job or the place rather than you personally or maybe he was just joking? Or since you have a crush on him and he possibly knows it because you flirt, he must protect himself.