I think that you may be afraid to defend yourself because you don't want to make someone feel bad. But you can't avoid that.

Sometimes you have to put yourself first because your needs are just as important as theirs. An important part of gaining self-esteem is realizing that. When self-esteem is low you are always worrying about what other people think or feel and you always ignore your own feelings. You even put yourself down so that others might feel good. But unless it is done jokingly and sparingly that isn't healthy and it's not fair to you either. Sure there are times when such an idea is useful but usually this is not the case.

You have to realize that you are important too. Sometimes your feeling will clash with others. But this is a normal part of life. We can't always agree with each other. What's important is that you make your needs known but in a non-confrontational manner. You have to say no sometimes and there's nothing wrong with that.