It is hard saying NO. But once said, it shouldn't take you long to feel that power of taking care of yourself. Sure, some will be angry and wel all know that they'll get over it.

I am the youngest of 5 siblings, two sisters and two brothers and I have very little contact or conversation. I found it refreshingly nice to skip the drama and all the stuff you get sucked into. Now that I say "no" I'm mean. Thank God for meaness.
Begin with those things you hold dear. Listen to your feelings then decide whether yes or no is appropriate.