Unless there is a witness and physical injury it's basically just his word against yours.

Sounds like he baits you so you react.