The good thing about having no kids with him is that you have no real reason to maintain any kind of relationship if you don't want to....

...I do second the idea of talking to a therapist to work through your issues....it might make you see things in a whole new way. Good luck!