It drives me nuts.

I do have a guess, but it involves "tall poppy syndrome," people who want to chop others down so that they stand taller by comparison. But any way I word it is going to come out sounding like, "Oh, they're just jealous of me," and I don't want to convey that.

Edited for typo.