It could just as easily be that he wasn't in therapy mode when you called, not that he was irritated. Maybe his mind was on something else and he just gave a quick response to your question, but wasn't dialed into your emotions? If I ever have a doubt like that after session, I try to spend the week thinking it's all okay. It has never not been okay by the time I saw her again.