I am going to guess, that it contributes to why you are the way you are now. Are you able to get angry with him? There were better compromises to what you ended up placed in a position to do, to soothe his fears(granted, it's the cheating and deceit of cheating that tears down the trust, but there's other ways, to bring back trust, that don't involve enabling him to feed on that fear). For that, I can see why your t wants you to get angry with him. I feel, healing this aspect of your life, which may take a bit of unravelling, will help you immensely, as you move forward. For instance, as him working through his trust, he could have asked that, if you were to contact the other guy, or to have heard from him, you'd promise to tell him. And another thing, could be that he'd be able to discuss any triggering moments, without repercussion, but in an adult manner. And if you'd done any marital counseling, to heal from this, one of the better methods of regaining trust are 'acts of kindness.' Not rehashing it, over and over. Not going to the point of invading all of your privacy. Etc. That could be the anger at him, that your t may want to unleash. Not necessarily the events that led to the affair, but the after affair behaviors.
You are a human being, with rights to be treated as such. That could be where working on the self worth, comes to play?
You could have been in a vulnerable position, that led to such an affair. I'm not sure, but either way...good to know, you are working through this. To bring the best self possible to the next relationship.