He assaulted your brother, and your brother was looking out for you... because of his actions, your family is now involved, and now you gotta listen to them. You still live with your parents.

But.. you know him best. You have been with him for 3 years and if you feel you can live with this guy and be happy, then it's up to you. Yes family can say anything, but it's only you who can make the decision, for yourself, not him not your family.

Either work on this relationship, communicate on what's going on with his mood swings, .you said you can handle it. Because arguments with family and bf/gf, drama happens. I've been thru all that, I got yelled at and got into arguments[not with my family but with his!] so, don't let this kill you, just calm down, consider you still live with your parents so to an extent what they say matters. Is it possible to move out?