I think this is a great opportunity to make your own traditions.

It would be bad enough now, but consider that making a tradition out of spending the holiday with your extended family will not be the environment you want to take your youngster to when they are older. I think now is the time to put your foot down and make the effort to avoid all that. Hence use this as an opportunity to start up your own tradition. It will not only be a stand in advocating for yourself but ensure future holidays are on your own terms.

Yes, it will be difficult. One excuse does come to my mind - the baby itself. You might indicate the child does not react well with crowds. I don't think it would be wrong to make a small fib and suggest the baby is fighting a cold.