Then let go. You have no reason to stay attached to him - it doesn't sound like you had much of a real relationship beyond sex anyway. Your needs are important, but they don't override his, and unfortunately both of your needs are at odds with each other - the difference is that he's handling his needs on his own, and you're wanting him to deal with yours.

You deserve so much better, and you're preventing yourself from finding happiness again by staying stuck in this cycle. Your child will always be a part of you... but you can live a good life still, to make them proud of you and to show that you did make the right decision for yourself.