I am going to quote from Sarahsweets (Everytime I set boundaries with people it makes me anxious) since she has a better handle on this than I do:

Reasons poor boundaries and anxiety go together.
So why does this happen? The subject of our example illustrates a common way poor boundaries yield anxiety. She felt as though she had to talk to her sister—almost as if she didn’t have a choice. Listening to her sister, making sure she didn’t feel disappointed, angry, or hurt (which the sister might have felt had our subject cut the conversation short) took priority over what she needed for own life—managing the to-do list and not overextending herself. She tried to make her sister happy and took responsibility for her sister’s feelings, blurring their boundaries. By doing something she didn’t want to do (talk for an hour), she effectively allowed herself to matter less than her sister. The result: anxiety. Does any of this sound familiar in your life?

I will add this though--you matter!!!! Every single one of us do!!! None of us should feel guilty about putting ourselves first. We can't be our best selves unless we do sometimes. You have nothing to feel guilty of: