Walking out on you is disrespectful. Of course you should confront him about it. I don't care if he's never done it before - that doesn't change the fact that it's a disrespectful thing to do and he really owes you an explanation. Perhaps there was an emergency or something but otherwise you just don't do crap like that.

It doesn't matter what he expects. This is a matter of treating people like they actually matter and if he can't do that...

I mean that's life. People say things you won't like. But if you don't speak up then nothing changes. Running away solves absolutely nothing.

Now this could all be a giant misunderstanding. However by walking away this could actually blow up into something completely different.