My ex abusive husband used to tell me I was starting fights all the time, when it was HIM who was starting them. Projection, which is another emotional and verbal abuse tactic. My ex was similar to yours. He was allowed to say "ouch" in response to ME, but I wasn't allowed to say "ouch" in response to him. It made me insane... very maddening.
So, what is keeping you in this toxic marriage? I describe it as toxic because those behaviors you describe in him are highly toxic behaviors.