I was raised you invite everyone or you invite no one at all. With that said though I realise that not everyone believes this so if I don't get invited it doesn't much matter. As for those invites you think weren't made so willingly, I sometimes feel more insulted at getting them than I would be had no invite come at all. The worst is when you are the second or last choice to be someone's "plus one". I think it perfectly okay to turn something like this down, you need not feel guilty.