I don't think most therapists would be annoyed by fidgeting. I am guessing they would wonder what it means, though. Is the client anxious? Do they want to leave? etc.