Well ... You need to be blunt with him about everything you have typed here.. There is one thing to be put in an uncomfortable situation and then there is being forced to routinely being in a situation where you are going to basically be ignored and they don't seem willing to speak a language you are able to join in with it?? That's just disrespectful to you...

Your concern is hurting him by not wanting to go .... But your going to be hurt by going with him and suffer through a whole holiday being excluded ?

If it were me I would just be honest and tell him you do not want to go, Maybe you two can agree to only spend X amount of hours with them and begin making your "own" traditions together as a couple?

If it were me I would start putting some or my needs and wants into the reltionship.. It cant be one sided .

I wish you the best