Didn't he try to end the relationship? On the phone you are the one who tried to continue it and yet you deleted everything, etc. I think, no matter how he tries to end it, what he says, it might not be okay with you; you keep giving yourself and him mixed messages?

If I were you, I'd write him a letter and end it however you feel would be best for you but so you know for sure it is over and not about him and what he does. You don't necessarily have to mail the letter if you don't want but getting it clear in your mind, what you want for you, might help?