Yes, that is a very tough situation. I don't think there is anything wrong with acting on your feelings. If you spend some time with your family and then reach a saturation point and want to leave, go for it.

I've found that limiting the amount of time I spend with my family at get togethers makes a big difference. I don't give them enough time to get on my nerves in the first place, if that makes sense.