I think it was confusing for you because he showed a part of himself (perhaps liking rough sex, being the alpha male, etc.) that you had not seen that is very close to the line with abuse. That what he did did not hurt and that he turned around when you said no and said he would not force you, I think you let the confusion take you away in letting him stay if you did not want him to stay. It's kind of like being an alcoholic and friends come over and bring liquor and you think you can resist but then decide, "what the heck, it's here, I'll go for it" and start again tomorrow.

I think you have to decide what you want and how you feel about things and then go for that at all times instead of letting him kiss you or "play" with your neck/chin, etc. You cannot have it both ways.