It's really hard to make these choices.

I like Hankster's suggestion. Could you do lunch with the family and then return home with your son? I think it is okay if you opt out of this, even if you have changed your mind a couple of times.