You know what, trust your instincts on this one. Touch does not have to be overtly sexual to be inappropriate. If you don't feel good about it, then something is setting off the alarm bell for a reason. Who knows, this therapist could very well be attracted to you and want to take this further. I am speculating, but our instincts are there for a reason.

Trust your judgement. You don't have to give any reason not to go back to him, maybe find another one, someone you feel more comfortable with.