I'm queen of not wanting to go/do whatever my husband wants to do so he goes himself and then I resent that, that I've been left alone :-)

I wouldn't take it personally. I think he's sorry that the "go away" slipped out and doubt you will hear it very often in the future. But the whole thing was about space and what was expected/looked forward to. I don't think either of you gave him enough time to respond to the surprise of you being there is all. But it wasn't about "you" I don't think, but about his looking forward to a state that suddenly wasn't there. I think he would have said what he said to anybody/his mother :-)

If I am in your situation in the future, I think I'm going to see if I can recognize the "problem"/surprise and "happily" offer to leave or ask permission to stay, etc. before my husband can respond to the surprise in a spontaneous way neither of us might like :-)