I am very angry at my mother for missing years of my childhood, those feelings are not something someone can just forget and move on from, although we would love to. So try not to make this about you (I don't mean to sound rude! Sorry). Yes he should have stayed with his dad if he is feeling so strongly about this, but I think on some level he did want to stay with you.
I still want to see my mom sometimes, regardless.
Try not to say things like "I don't want to push you away" because if you put yourself in his shoes, that doesn't sound very good. My mom has also said some things similar and it made me feel like she was using it against me as an excuse to continue being absent and she really didn't want me around anyway.