I know how being afraid/anxious one's self can trigger fears in others or make one look as if one feels "superior". Just not joining in/talking can do that, make other people uneasy because they don't have any way to "know" you and make a judgment of what sort of person you are; the unknown is almost worse than the known threat sometimes.

Somewhere in the back of my mind is the concept of "consistency" too. If one's verbal expressions wander all over the emotional map; sometimes kind, sometimes judgmental, sometimes angry, sometimes gentle or thoughtful; the other person isn't sure "which" you they might get next.