If I were you I would leave after dinner. And you don't have to feel bad for leaving, it has nothing to do with anxiety, but with the fact that going to a binge drinking when you don't drink is like going to a married couple meeting when you're single.

As for me, besides having anxiety, I'm very introverted and I really don't feel like going to most of the social meetings I'm invited to. People seem to care about attending to places a lot more than I do, and I honestly don't understand why it's so important to go out in weekends and do something every week. Sometimes I'm overwhelmed with the thought that I won't be invited anymore and how hard I am making my own life, specially in the rare occasions I decide to attend to and end up hearing what people say about who is always missing social events.