Sorry to hear that. Can you pretend to be sick a day beforehand & even go to the doctors to make your excuse convincing?
Maybe take an edible or have a few drinks if your husband forces you to go there with him?
Go in separate cars & pretend to have a severe headache soon afterwards & then leave.
Who cares what they think? It’s not like you see them often & you obviously don’t want to see them either.