It's also about personal boundaries. You have a right to set clear boundaries without ever having the need to explain yourself.

Even if he wasn't aware that it was wrong (which seems strange since most people intuitively know that going onto someone else's phone is a gross invasion of privacy) the fact that you've told him to stop and he still does it is a huge red flag.

You never have to put up with that kind of behaviour whoever it's from, friend, relative or partner. Anybody who tries to put it back on you is playing mind games. I had a friend once who I caught looking through my phone; when I called him out on it he tried to make out I had the problem and that I should have put a PIN lock on it. I'm not friends with him anymore as it was the latest in a long line of humiliations I'd passively tolerated over the years, all through trying to put other people's needs ahead of my own - something that eats away at self esteem like nothing else.

I don't know about ending the relationship as that's a decision only you can make since you know all sides of him but I would advise going back to the phone thing and making it absolutely clear that there is no discussion to be had - he must not go into your phone. See how he reacts and I'm sure that will guide your decision on whether to stay with him or not. You haven't mentioned it but if there's even a slight chance he'll get physically aggressive, do it while other people are around.