Your family is not going to like it when you set your boundaries. Be prepared for that. That is pretty typical. They will try to push back usually. But it sounds like they aren't particularly great to start with; remember that, too. The difference with setting boundaries is once they've tried to push back a few times and you don't budge, they'll eventually give up or adjust in some way -- often with more distance, but that isn't always a bad thing.

As far as your niece goes, find other ways to interact with her. Make those encounters of short duration, preferably with others present, maybe in public places where she might keep her behavior in check a bit. Overnighters, long stays, one-on-ones where she will feel more power to say whatever she wants are situations to avoid. Keep boundaries on what she is allowed to talk to you about; you can tell her that you won't discuss your mental health with her. You are the adult in this relationship and it sounds like she needs some adult parameters.

Developing healthy boundaries for ourselves is initially really hard and can meet with resistance, but over time, it really will pay off for you in greater confidence, less anxiety, more of a sense of autonomous self. Keep us updated.