I think he's trying to hurt you on purpose because he's still hurting and hasn't let go of the past. Its sad you weren't there for him much during his life. But if you've apologized and have tried to move forward with him in a better more communication fashion and he hasn't accepted it. You may have to give tough love. Try if you can to initiate a conversation with his father and possibly later include him so he can understand why you weren't around as much as he wanted. I that doesn't work then don't allow him to use you because you are feeling guilty. Tell him you love him if he needs you you'll be there if you can and if you can not you just can't. He's an adult and sometimes we can't allow not even our children to hurt us.