Is it worth bringing it up with the therapist?
Eg "sometimes I get the feeling that you're annoyed with me. Are you? When you say X I feel like you're impatient/irritable/whatever. Also I get the feeling that you'd be angry or stop seeing son if I terminated therapy with you. What are your thoughts about that?"
I think it's inappropriate for him to be angry with you or make you feel like you can't stop whenever you want to. But I also think it would be helpful to have him say that in so many words. Sometimes we attribute our own issues to our Ts and we need to give them a chance to say what is really going on for them. But of course sometimes they really are working out their own issues on us...
Could you print out your post or this thread and share it with him?