I think it also comes down to boundaries that we set for ourselves. It takes time to develop this skill of assertiveness, and it is a skill. Start with small things, like Kaliope said. Then get to the bigger stuff as the skill develops.