Probably, rather than guilt, you are feeling shame. Guilt is a feeling about something specific that you did. Shame is a feeling of "I am wrong/bad."

Shame is an inverse form of aggression. A learned (but a *passive*) reaction to aggression.

Sounds like you need to practice assertiveness and setting boundaries.