Obviously I don't know your boss so don't want to assume either way, but is it possible he simply asked without thinking, out of concern for your wellbeing - and it didn't occur to him that actually it's not his business? As you didn't mention him making a big deal out of you actually going so maybe he literally just wondered? I am the same as Sheba, I can get paranoid and assume the worst but sometimes it is just a misunderstanding. I hope that's the case with your boss!