I know it's very hard but you have to stop LETTING them control you.
There has to be a balance to it all. If people sense an emotional weakness you will be taken advantage of. The emotional part of your brain will not let you take control of your life. Use your reasoning ability. That part of your mind that is more "realistic ". Just do what you feel comfortable doing and leave the rest.
You can't put $5 in the collection plate if you only have $2 in your pocket.

Good luck to you...