Marriage. or any relationship, takes effort. No one gets to decide how much effort the other person puts into it. All anyone can do is give their best effort in any given moment. Some people's best effort may not match our own.
I've never regretted giving someone else more grace, kindness or understanding than they deserved. It's only been in the last year or 2 that I've been able to do this with the abusive ex I have. I understand how difficult it can be. My only suggestion is to not say words you can't take back. Discussing this while one or both of you has emotionally charged words swirling around isn't going to help. Agreeing to sit in silence until fears and anger are under control is still agreeing. A half step forward after a 3 step backwards stumble is still forward progress.
As for feeling bad about his comments, neither one of you married stones. You are people who grow and change everyday. It's logical to say neither one of you are exactly who you were when you got married. It isn't fair of him to use it as a weapon, so it makes sense that you are putting up a defense. I think a couples session may help you both. If he doesn't want to, an individual therapist chat may be beneficial so you can find a better defense than awkward silence and brewing stressful reactions.
I hope you find a way to resolve this conflict.