I don't know, Ruby, is it possible he was joking at all? If I'm going on vacation the last thing I'm going to think about while I'm gone is work or my coworkers. And if someone said "I'll miss you while you're on vacation" I'd be like "I ain't gonna miss any one! I need my vacation!" I could see this being said with a humorous intent.

Regardless of humor, I don't think his intent was to make you feel bad. But he's your supervisor, not your friend. It's likely he keeps a very strict boundary between work and home. You need to respect that those are his boundaries.

Also remember that what people say is more about them than you. Try to remember to not take passing comments personally or let them be about you and remember that what people say is often a reflection of their own internal state at the moment versus having anything to do with you. It might be only 25% to do with you and 75% to do with how he's feeling in that moment.