hi sherwar,
I can understand how hard it is for you to let go, it's never easy. Our instinct is to do whatever possible to keep them right next to us. The world without them seems so uncertain and scary.
You strongly believe that you cannot let him go and eventually he'll change his mind and end up staying without being forced. I do believe, that even though right now you're not ready to let him go, you will be able to find that strength as time goes.
Like everyone says, with time you'll heal if you let him go
With time, you'll find the strength to let him go too, on your term when you're ready
I agree with everybody that it's best to let him go if he won't love you. It's his loss since he has you and his son who love him very much but he's willing to throw that away.