If he says he doesn't want to go, tell him you figured that would happen. Tell him dinners in the freezer (or there's stuff to make a sandwich in the fridge - whatever you have, even just a can of soup) then leave. Go be with your family. If anyone asks where he is, tell them the truth... he chose to stay home. He makes his choice and you aren't responsible for it.

Accept his refusal to attend with grace. Imqgine how confused he will be when you hold yourself together and plan ahead for his normal behavior. No one will be surprised or upset when you tell him things are over for you (whenever you choose to end things).