Abuse takes many forms, from the outside looking in I'd definitely say it's abusive. He may not be hitting you but he's threatening to hit you. You can't say one second "I'll beat you" and the next second say "but I'd never hurt you", he's keeping you on unsteady ground rather than providing a firm base for you to stand on which is what a partner should do.

If you've noticed that his personality is someone who "likes to fight", chances are it's only a matter of time until that gets turned on you, and it seems he's already turned it on you psychologically.

From the small amount you've divulged it seems like a pretty dangerous situation for you, he's a ticking timebomb. What are the positive things that draw you to him?