To me, your friends and aquaintances don't sound that far out of the ordinary. Some are going to push the envelope because "if you don't ask, you don't know." But it sounds like you may be holding your boundaries when you say "no" to them. Sometimes you can, sometimes you can't and maybe it's them that need to learn that. As for accepting them the way they are, that's a good thing. Next time they ask you why you don't, ask them the same question in return.

You are the one that teaches people how to treat you. There's no need to withdraw unless one or two in particular really tax you.