No, you are definitely not alone. I also don't have social anxiety, can talk to strangers and am generally fairly confident. But I can't stand having my personal space invaded. It is so bad that this is even true for my husband and son. I love a hug and all that, but if they stand too close to me for too long, I move away. Hubby and I don't even sleep holding each other (well .... all that often).
I'm terrible in a queue at the store. If someone comes and stands right up behind me, I give them a dirty look (

) and move away. I'm awful in crowded places, elevators, escalators and the like. And just let somebody I don't know touch me - I'll have the heeby jeebies for weeks.
I get terribly claustrophobic with people too close to me. And that includes unwanted or uninvited communication (texts, phone calls etc). Makes me feel terribly unfriendly sometimes, but my family have come to accept it.