I'm curious how you can go off with him on non-therapy outings if you are afraid of him.
It sounds like a confusing relationship to me. I wonder if the itimacy in the real therapy setting is confusing and feels like too much because of extra-therapy contact.
Maybe if the relationship becomes one that is limited to the therapy session, it might feel safer and more secure. It might be worth a try to see.
Yes, mention it. It's important.