Is it just with people or do you suppress anger and annoyance in general? I know some people who have difficulty expressing negative emotions or being assertive. There was a Simpsons episode where Flanders had this problem (sorry it just reminded me)...and in that case it stemmed from not being allowed to express negative emotions as a child. That's just one possibility though.

I can see this being a problem for you and I wish I had better answers. Good luck.