Your boyfriend is in his 40s, and can't have a conversation about you not visiting his family without getting angry? I'm sorry Hvert, I really don't mean to beat him up here, but it just sounds like he's really immature, and making it unnecessarily hard on you.
The way I'm imagining it is that you try to bring this up, and he gets mad. How do you guys even begin to explore a compromise like that?
I still say "don't go, let him be mad." Everything you're writing about his father and his family is reinforcing that, in my opinion! I mean, the holiday is for you TOO. If this were just an hour or two drive and one dinner, I could see sucking it up to keep the peace, but for that many hours in the car, and that many days... it just seems really unfair that he's not willing to work with you at all on this. I'm sorry
I don't suppose it's possible for you to get a hotel room, and tell him he's welcome to stay with you or at his parents, whatever he wants?
Oh, and good for you for telling him you won't be cooking!
Um, and I know you don't want to stir up more trouble - but if his father decides to go for a drive when drunk, can you call the police? That's really *really* scary, and he could end up killing someone

so not worth the risk.
Sorry... holidays should be a time to relax and enjoy yourself, it really sucks that you have to deal with this every year, and that your boyfriend isn't willing to work out some sort of compromise.