I think it’s a slippery slope, to be honest. From what you’ve written he is crossing boundaries he shouldn’t as a therapist. I’m not blaming you for the situation you find yourself in, though. It’s his responsibility to uphold his professional boundaries if you or any other clients bring sex/intimacy up.