You said, "All the torture he put me through for almost a year." If I had read nothing else in your post that says it all. I think you're right to cut off communication and if it was me I would hire a lawyer (if possible) and make it so he cannot see his daughter. I know it sounds harsh but if this is the bs he pulls with you how will he treat her? It seems that he has managed to damage your self esteem so you feel responsible for his actions. Do you want your little girl to grow up feeling the way you do now?

I've had friends in similar situations that did get married. They're now divorced and raising their kids on their own. Why put yourself through that heartache? I agree with Nola22, stay home - don't go see him. Work on a happy home environment, one that you and your daughter can both flourish in without his interference. Most of all, cherish your child and yourself.