Are these people actually saying this is annoying to them or could they just legitimately be asking why? To me this actually indicates they are interested. After all, if they were disinterested or didn't care they wouldn't be saying anything at all.

Of course, your reply could always be, "Why do you ask?"