Irritated, no. Frustrated, yes. I do see the two differently.

I have always found his frustration to be completely justified.

I have had no problem getting past it. I've always been able to see it as pointed specifically at a behavior, not a statement about me as a person. Can you maybe look at it as a specific moment or situation than a broad comment about you as a person?