You're right to be annoyed. It sounds like his own impatience talking, and he really needs to own and examine that countertransference, and not act from it.
No wonder it's hurtful and inaccurate, he's not meeting you where you are.