I honestly think that you have to make that decision when enough is enough, and screw what everyone else thinks. I did that with my ex, and he was verbally abusive.

I got tired of being alone in the marriage. He was a charmer, though, and when I left no one understood why I'd do such a thing. And sadly, people do choose sides. I lost many friends, not to mention that side of the family.

But you have to do what is best for you. If you're that unhappy and that fed up, then it's time to make a change. Life is way too short to be so unhappy...