I have a really hard time asserting myself socially also. But oddly if someone I am comfortable with invites me out I will go and be mostly O.K.
So I agree that being familiar with the people is a big part of it. I would have all kinds of anxiety if I had to go to social gathering and didn't know anyone.
Hang in there. We don't stick out in the crowd as much as we think we do.