The one thing I always wanted my mom to say was that she understood and was sorry, without justification. I have no idea if this is the same when it comes to your relationship with your son though, because for all I know you could have already done that.

I think its best when he starts acting hurt and saying those things, to put yourself in his shoes and imagine what you would want your mother to say to you. You could say things like, "You're right, I'm really sorry, I shouldn't have done that. I'm sorry for hurting you and for not being there for you."
Saying anything like "I dont want to push you away" implies you are pushing him away but you don't want to, does that make sense? Anyway...just say sorry whenever this comes up again, because its not like you guys can rewind and go back (don't say this to him either). Basically don't say anything that would come across as cold or justified or like you weren't in control of your actions - these things won't help.
Taking full responsibility in your words and not making the situation about how hurt you are, is the best and probably the only way you can get through to him and past his anger.