Personally, and this is just me mind you... I would have told him in the moment that I didn't know he was waiting that long and ask him "why didn't you text or call or something if you were here already?" (Assuming he did neither of those of course, because you didn't mention that he had.) I don't like passive aggressive BS and him acting that way is awful childish, I always call people out on it if I catch it (but my social retardation does not let me catch it as often as I would like). Maybe he was having a bad day or something, who knows, but you are absolutely not at fault if he didn't give you notice and he should not be taking his bad mood out on you.