I feel like he's playing a mind game. "oh, you want off your meds? Okay, you must not be that sick then. You just have problems adjusting to college.". My guess is that he disagreed with your decision to stop meds and that led to his inappropriate behavior.

Just a theory.

Good luck. Hope you are back with someone decent soon.

EJ