I know in religious communities people will tell you to pray when you are in a difficult situation, but I do not think that God wants us to live a miserable life, he wants us to be happy.
Just to bring up one point you need to consider-if he wants to be in the child's life when he takes your child on visits for the night or weekend you lose control over what happens. Some people stay until their kids get older so the children can vocalize what happens at the other parents house, and if there is drug activity, abuse, etc. they can talk about it to the proper people. It's really hard to terminate parental rights, even convicts can get parental rights, so please consider that in your decision making process.