I think you're being very hard on yourself saying that you "should have set boundaries". That's a process, not an on/off switch. It takes time and you kind of have to build it up - the boundaries themselves and the skills to maintain them - over time.
You have STARTED putting up boundaries, and that is BRILLIANT!
What I would do now is treat this as a learning experience. Think about, say, your conversation with your mother, what was not okay for you, and what you would like to change. Maybe make a list of the things that were not okay. This is the first step to training your brain to help you maintain boundaries.
I know it's hard but it will get easier