I figure I annoy everyone at some point or another, so I don't fret about it too much. I figure therapists can handle a bit of annoyance from time to time; seems like it would come with the job. That said, unless a person really talks to me about what I am doing that they are finding annoying, then I assume they are handling it and not all that bothered by it either. I mean, we all get annoyed with people and are able to let it go; it isn't like it's the end of the world. If there is someone in my life who I am finding my level of frustration is at a point where we need to discuss it and come up with a plan, then that's what we do (I'm a high school teacher, so there's "normal" annoyance and then there's annoyance that requires an intervention so to speak). Assume your therapist is handling whatever is going on and that he is professional enough to let you know if there is a problem.