I'm annoyed because it makes me feel vulnerable, dependent, stupid, desperate and so on. I'm not sure exactly why, but that's the way I feel. Plus, I hate the fact that my English is still full of errors.
I don't think I'd like to be the centre of the party even as a NT. It's just not me. It's quite confusing. I like to be noticed when I'm doing something good (not all the time though) but at the same time I hate if all the eyes are on me and/or if I get too noticed.
In a situation like that (going to an event) I feel comfortable to talk only with people I've known for quite some time and with whom I get along well, of course.
I too take no initiative in getting to know people, but, if they seem like nice fellows and they talk to me first, I may feel comfortable with them in the future.
Other reason I don't like to be seen inside a building is one the rules of politeness: say 'hello'. I don't feel comfortable saying that.
It would be perfect if 'they' hadn't interract with us at all, but instead waiting for us to ask for things.