He "[...] never gets violent with me, insults me or threatens me at all" yet he berates you, yells at you and gets into these screaming matches?!

And no, he doesn't have the freedom to "verbally berate" you and expect you to take it. You are allowing him, and giving him permission, to yell and verbally berate you.

The difference here is that he placed a limit on what he finds unacceptable (don't hit things) whereas you are not placing any limits on how he treats you and accepting his verbal violence.