I hope that you can work something out. Everyone is correct that he (your bro-in-law) needs to respect your wishes but with your husband undermining it after being made to feel guilty it's a tough thing.

What comes to mind is that somehow maybe your husband could find ways to spend time with his brother outside the house as much as possible, if of course he can't stand up for you. Although I get the impression you're not wanting to make your husband feel guilty, no matter how you look at it, you are supposed to be his #1 priority, over friends and even siblings. period. Not something that you can easily voice to him, I understand that but still is the truth. That's why I think your husband ought to do something to give you space as much as possible if your in laws come by and stay.

I can't really think of any other solution..

Do talk to your husband more about this. somehow make him understand that you'd apreciate him standing up for you and not sacrificing your needs due to guilt. His brother should be 3rd, after you and your kids.