I wonder if pushing him despite his distress is what's effecting you more than the fact that he is easily triggered...
One of my dogs seems to have developed severe anxiety symptoms the last few years. On the one hand, I want to show her that the things she is fearing are not going to hurt her, but I can also really relate to her extreme anxiety. One therapist pointed out that I was pushing her past her limits in ways that would make me a huge mess if someone did the same to me (even if I trusted them). I started changing the way I interact with her around the anxiety. Unless absolutely necessary, I only push her as far as she is willing to go with things. I try to take time to reassure her when she starts to panic (nails are something she reacts badly to, always had)...

Now I'm able to find comfort in having someone around who understands my own symptoms, even if she can't verbally express it. It helps keep me in check to a degree also - I try to minimize my anxiety around her and use better coping skills so I don't cause her any extra stress (her anxiety started shortly after I attempted suicide and the dogs were around. She used to only panic if things reminded her of that event, but in recent years, the anxiety has grown to be triggered by a lot of different things).