I find that compressing my emotions sometimes makes them much more uncontrollable when they surface - and they can spring out when they choose. Perhaps, each time you are done with an encounter with him, you can go in your office or the lavatory and do really fast 50 jumping jacks (releases pressure). Then you carry your annoyance less, and the exertion will create happy endorphins. Next time you are talking to him, you will think of the jumping jacks with pleasure and also - you will know there is a release to the flow of anger and feel less trapped. My 2 cents. Best of luck!

edit addition: I struggle with fixation too.