I think the important thing here is that his actions are making you uncomfortable. If your therapist says or does something that makes you uncomfortable, you should absolutely tell him that. Even if he's not doing something objectively "wrong" it's important to explore why his actions affect you this way. I don't think you can make much progress in therapy if you don't feel comfortable with the therapist.

I see you're worried that he'll terminate therapy if you bring things up, but if your choice is between termination and therapy with a guy who makes you uncomfortable and can't handle talking about it, you are better off terminating.