Thanks to everyone, this has been a really valuable advice
I'm getting that bring physical violence won't do any good, as it will probably just make things worse. And I get that he's looking for a reaction from me; I see him pushing buttons.
I have a nephew and niece, then I was hoping to stay in contact with, but I have the option of going no contact with my brother, and I may actually take that.
He behaves this way at the expense of his children having an uncle and I've decided so be it.
For posterity, I'm considering sending his wife a message to let her know why I'm not around for the kids. I'd like to be clear and why I haven't been around. thoughts on that would be appreciated as well, if any.
Bottom line: I have my own problems and I'm not really going to waste anymore time worrying about him.