You have time invested in this but you really need to look at the whole picture..

Are you willing to give up the idea of having a child? will you regret it later?
People get hurt and cheated on, his refusal to love again seems extreme to be honest.
You and him have been together for 4 years... what binds you together? yes you love him.. but why does he stay? that is something that you really need answers to, i think at least.

Good luck in whatever you decide.