Yeah, even if you had annoyed him it is still not your fault. The only instance I can think of where it would have been acceptable for your T to leave like that is if he felt threatened by you physically. Obviously not the case here, so that makes his behavior unacceptable in my book.