I think that question is the perfect gauge of a relationship. You should always feel safe. When I was with my ex I would "forget" the times he would beat me up, belittle me in front of friends, accuse me of things I didn't do, etc. just so I could have some semblance of "safety".

Even if he never hurt you physically again, would you want to live with the fear that he might?