Neither of you should be laying hands on the other in anger. I don't know what you mean by he doesn't let you walk away; I would explain to him you need and would like the time to cool down so you can discuss whatever calmly. If he grabs you or gets physical, that's not acceptable.