I would guess that you are feeling guilty and the "shoulds" are working on you.

Go with your gut feeling - you have to be true to yourself and your well being. I limit my family interactions with those that make me feel inferior, bad, etc.

Give yourself permission to make a choice that is good for you. It is okay not to go.