I dont except or decline invitaitons based on retaliation. I do so based on if the event interests me, whether I have the time, finances and whether my wife and I an find sitters for the children. if I decide not to attend I am completely honest with the host of the event...

Im sorry i have other plans
Im sorry but Im just not interested in going to this kind of party right now. Im feeling a bit like I would like to stay at home and ....
Im sorry I am having problems finding a sitter, everyone seems to be out with their friends and family doing their christmas shopping and partying their selves.

by being completely honest the host sponsoring the event sees that my attendance is not a personal attack on them, which means they have no reason to retaliate by spreading rumors and such.

my suggestion is make your decision of whether you want to attend or not based on your own interests and such then be honest with the person with why you dont want to go.