It's not just a matter of letting go, per se, but looking at where there can be compromises. With his attitude of wanting to irritate you more, rather than displaying an ounce of compassion and desire to meet somewhere in the middle, leaves me wanting to ask, is he seeing his own individual therapist? Sounds like he could use some letting go work and learning to not harbor resentment, the resentment that leads to his desire to irritate you.
Perhaps, making a list for yourself. The unbendables, the areas that can be compromised, and what can be taken off the metaphorical table/drawing board, can help?

And, yes, I felt like this with my exh.