I think I know what you mean. I am a socially anxious and shy introverted person and quite often I just dont't want to talk at all or even find the company of people stressful.

This is so easily misinterpreted as being grumpy or angry or perhaps having an antipathy for them.

So you could perhaps trying to explain that to them. You don't have to be a downer if you tell them this.