It is honestly difficult to give advice without hearing your children's perspective. Clearly there is a reason your son reacts the way he does. Generally adults do not just go off what someone else says, they form their own opinions. It sounds like he is trying to express his anger and hurt to you, be it not in the best way, and you are meeting him with a solid wall of saying you didnt do anything. If making amends is important to you, I would suggest getting a therapist to help you work through it.

As a side note, I didnt not have a good relationship with my parents. Had they got a lawyer involved to see my kids, I would have cut them off permanently. No parent is going to let someone they find to be toxic around their kids. I encourage you to really listen to your son, open your mind and consider both sides. Also an impartial therapist will help with this. Good luck.