Yes I agree that is a VERY inappropriate statement for a T no matter what he meant by it.

IF you otherwise trust this guy I think it is worth clarifying. For example maybe he meant it as an off-the-cuff joke, one of those things that trips out of your mouth and then you realize you shouldn't have said it. Or maybe he has some issues there and it was a joke borne of a Fruedian slip?

I still think a T should know better to watch his language, but we can all make mistakes, and if it is worth it to you to continue this relationship (patient and therapist relationship... I didn't mean something more!) you should let him know it stuck with you and clarify what he meant... if he meant anything at all... by it.