One of the few things I know from experience is that not doing things makes my world smaller. I got my world so small that it was all inside my head and not a great place to be. It took me 25-30 years to get out.

I would ignore the feelings of wanting/not wanting to go and just go, just experiment with being there and seeing if you can connect with one person or enjoy yourself (not think about yourself?) for 5 minutes (I suggest playing the badminton or softball :-) arrive early or late (as you prefer; I like early before there are so many people so I can get the lay of the land) and leave after X period of time, your choice (or wait until you really "feel" like leaving; I've been surprised to be enjoying myself and stay longer than I planned).