If this shithead ever did go too far, which it sounds like is most likely, you could have a hard time pressing charges because of your history with him, ongoing relationship and communication with him. Sad but true. And if he got away with it with you, guarantee he'd do it again to someone else. This needs to stop.

Protect yourself. His feelings be damned, he obviously doesn't care about yours. Stay away. You are worth SO MUCH MORE than this!

(This is coming from someone who has personal experience being sexually abused as a child, not knowing how to say "no", starving for affection from men, everything you're talking about. I paid a personal price, and continue to every day, for not saying "no" soon enough and getting away from an unhealthy/dangerous situation.)