PS So basically what you need to tell him is this:

"I ruined your life? I did not mean to, but if that is how you see it, then fine, let us just make sure I do not do even worse things to you, so let us part company asap. And, btw, no, you do not have any right to be informed of whom I choose to talk to."