That is NOT okay for him to say! When I disclosed a similar incident to my T, she made it clear that only a strong "yes" and while sober is consent. Saying no or not saying anything at all is not consent. I'm sorry that your T was so callous about it.