please definitely don;t answer his call if he does do that. You are correct that you do not need that. I am soryr he put you through all that. It is perfectly ok to cut him out of your life. Can you block his calls and texts? I recently did that with my dad's numbers. It helps a lot. Everythign goes to vm and I just delete it witout listening. I think it was a setting on my phone. I have to keep him in my contacts, but there is a setting that will block his numbers from coming through.
Good luck and try to stay safe...