I'm pretty stressed out and for the most part trying with every part of me to keep myself from doing something stupid to this man that I will regret.

It's obviously an unhealthy situation for everyone involved. You need to figure out a way to get out of that house and make a life for yourself. Maybe you and your brother can get an apartment together.