My step dad wasn't nice about me when I got unwell at 18.
You know one thing I heard him say was "i wish she had gotten ill sooner so we could have put her into care"
He called me "a pitiful human being"
You just have to learn to not let it get to you. It used to make me incandescent with rage I would simmer silently. I turned my anger into being driven to succeed. I took up running, read hundreds of books, quietly plotting till I was strong enough to go back out into the world.
I learned to forgive them. But I fell out with them first. Was completely alone. Its ok to cry but you have to stay strong.
If your brother has issues too it reasons to say that maybe he wasn't the best parent he could have been.
You seem to have developed a strategy, by just seeing him when you have to....so at least you have space.

Please don't take any drastic action. Find something that keeps you going until you find help, something that you enjoy, a release, an outlet. Or just reach out on here....your doing the right things