Just don't go. Bipolar patients can't afford to put themselves into strong triggering situations.
What's wrong with saying that it's just too stressful for you (and probably the children, too?)
I know a fine person who used to say that going to see in-laws was like a visitation. If it
appears inevitable, see a psychiatrist and ask for medications to calm you during the day or
days you have to be there to show respect to your husband's wishes.

Does your husband have any understanding of what his family members are like?
If he really insists, I'd try to make sure you have private accommodations and that you
can all get away from them in a hurry if the negativity raises its ugly head (like the
postman in "Funny Farm"...).

I think it's just too stressful to have to deal with things like that. I frankly just can't do it
and won't do it. Eventually, the talking stops and life quiets down into a stable,
pleasant lifestyle--that is, away from that sort of thing!

The problem is primarily one of ignorance and how to help correct that is just numbing. It's better not to deal with it at all, in my view. Some of us just have to stay away from the problems of others that can trigger us.

I wish you quiet days, soft nights, and peace at the end of each day.