He has gone against your expressed desires and wishes twice or three times. You communicated it bothered you. He should respect that. He is not respecting you by doing this repeatedly. Respect your own expressed needs and don't go back on your words now. You told him you would leave if it continues. My advice? LEAVE. It seems he won't stop on your behalf or out of respect for you.