Why not just say the truth (or part of it?) You say others don't go, and you give them the excuse of not going because of drugs or something... but who knows for sure? Why not just say, No, I won't be going to any of the family get togethers for a while? No excuses, no lies. Just like here at PC, if you state it as a personal statement and just put it out there, they will hear it.

They might not like it, as abusive ppl tend to want to control everyone. But no matter what you say (and have said) I bet they talk about you anyway, and don't believe anything you say anyway. Sorry, that's very negative of me to say that, but it could be very true, you know?

Tell them short and sweet, No, I'm not coming. And when they ask why just keep saying, I'm not coming. (It is up to you you know?)