I think he was more implying that he won't be thinking "about you" or any of his colleagues, or his job.. because he's on VACATION! I've heard people joke in this way when taking off for a much needed vacation. I don't know what your relationship with him is like, but it sounds more like an innocent joke about his vacation.. and maybe you personalized it.