Maybe you can try having a converstaion with your Mum about what she would do if she felt threatened by your sister. Like, who she would call or where she would go if your sister threatened her (a neighbors, good friend, you). Since your Mum is an adult you can't really make her do anything and the way you are honoring her privacy/wishes sounds appropriate at this point to me.

I know for me that growing up in a highly dysfunctional home didn't give me a clue about boundaries. I've gotten better but I still don't always recognize (especially in the moment) when my boundaries are being crossed because the uncomfortable was my confort zone. Now, I'm always asking myself how my boundaries feel and if I feel ok with a situation/person.

If your boundaries are strong then your sister will have less power over you and while you can't strengthen your Mum's boundaries directly you might influence them by just being yourself. If you google boundaries there's a bunch of good stuff online. I often google boundaries just to remind myself of my goal to make mine healthy!

Family stuff is so intense and loaded with history. I commed you for trying to make some sense of it and caring for your Mum.
Good luck!