Well he did not call the police so don’t hang on to the fear of that or the what if’s. Take what his reaction was as a warning so you don’t react to him in any physical way.

Sounds like he likes to test pushing your boundaries to see if he can get a reaction out of you.