It sounds like you are feeling threatened by the presence of others, and hyper-vigilant.

I wonder if it might help to distract yourself while out, for example focussing on something else - maybe buildings or the lanscape around you. Have you tried listening to music, preferably a type which is calming or feel-good?