How many times has it been about you? If they were whispering about you, what are the chances they would tell you the truth? I guess I'm getting at the point that asking the question doesn't serve you well because they'll likely tell you no anyway. It will get easier and bother you less the more often you refrain.