My opinion: don't let the past ruin the future. The relationship didn't work and he has treated you like crap oer and over. Now, you are allowing someone to be in your life that is causing issues with a relationship that could potetially go somewhere. I understand that sometimes ex's can be friends but often times too much risks are involved and there is no reason to keep someone around that is doing you no good.