you feel guilty but the fact is that is what he is using to manipulate you or attempt to. You need to block him from contacting you and if he comes around you need to put out a restraining order on him. The only way your guilt will subside is by eliminating the thing that is exacerbating your problem and that's his manipulative words. Like Bill said, it's emotional blackmail and the only way to get rid of it is to not listen to it. If you keep the lines of communication open, likely you will keep falling for it, so best to just cut him off, change phone numbers if you have to but cut him off completely.