That's rough. Do take care of yourself. Do you think there could be any positives to them spending some time with their grandmother? It does seem a strange thing to do under the circumstances, and could be really dangerous for her!
It sounds like a really complicated and painful situation.
It sounds like you get on fairly well with your brother too? You said the boy's mother is emotionally abusive. Do you think your brother and his kid's are being abused? It's just usually the case that family/couples therapy isn't effective when there is an abusive dynamic. If she isn't getting any help for herself, and it's affecting the rest of the family very badly, it could be the case that only distancing from her will make things better. That's not always something you can promote easily either. If this is the case, I would really take care with what you say to, and about her. It could actually be a good thing that she is scared of your nephew.
In my experience abusers do not have healthy shame. They do not respect people. They see people as objects that are either useful, problematic (solveable), or best avoided. If she is pushing her son away rather than using him or "solving" him, then that is actually the most positive option, unfortunately. This might all be way off base...I hope so, but guess not