I'm so sorry you are in such a difficult position. I can understand the guilt you might feel about your brother given his problems. But if you have such negative experiences when you are with him it's probably not a good idea to put yourself in that position with him again. You can't help him if you are a mess. You need to do what you need to to be safe and strong. Is he the kind of person you can talk to? Is he able to listen? If not then it's not your fault. You can't fix him.
Maybe you could talk to a counselor of some kind and get some specific ideas on how to cope with him. Maybe the relationship forum has some ideas.
In the meantime take care of yourself and post here whenever you need to.
Good luck