I do this. Sometimes I even get flutters of anxiety in my chest when my own sister drops by unannounced with my niece, and we grew up together! Whenever the gardeners are outside I dread having to hand them the money, and my heart jumps into my throat when they knock on my door - and they come twice a month! Out of all the towns I've lived in, I've never known neighbors to stand outside and talk to each other, so I don't think that's what neighbors really do unless they're already good friends or something. It doesn't matter if your neighbors think you're stuck up because neighbors probably have their own busy lives to live and aren't really thinking about why their neighbor didn't say hi to them.
Even the FedEx man freaks me out when I have to sign for something at the door, or when I accidentally run into the mail man while he's loading the mailbox. I never answer the door if it's someone I don't know - I've heard of too many scary stories about just answering the door to anyone so it's not worth it to me. Thank god for the invention of peep-holes. I never answer the phone either, if it's important they'll leave a message lol.