The fact that he woke you up to threaten you, is enough!!!

I don't know what he threatened you with, but that would be enough for me!
I don't like threats.

My ex used to do that to me to make me stay. He always threatened to take my kids to somewhere where I'd NEVER see them again, and I KNEW he was capable of doing that, so I stayed. I was so unhappy -- I waited until my youngest was 18, and then divorced the bum. My kids both asked me why I waited so long -- it was hard on them too, but all those years I didn't want to tell them that their father was a @#$@. Unbeknownst to me, they figured it out themselves. LOL
If you don't have children involved where you'll have to fight over them, I'd pack up and leave!!! God bless you and take care of yourself! Hugs, Lee