I have a total care-taking compulsion so I understand feeling like you have to do this but it sounds like your brother is literally draining you. He is taking a very real toll on your emotional well-being and your marriage. I really think you should set some boundaries, setting a time-limit was a good idea.

One way my parents eventually found for getting my uncle off of their couch was they would tell him the terms of crashing at our house was he had to do some minor job around the house. He would nail a couple of boards down and be gone the next day. It was totally reasonable and completely effective.

Figure out what totally reasonable thing he would never in a million years be able to conform to, a curfew, no drinking in the house, etc. and tell him that's the new house rule. If nothing else it could be something that makes your husband happy.