Honestly, sometimes I just don't answer the door. It may seem rude, but then again, I think it's rude to come over unannounced and it stresses me out. If I feel up to answering the door, I do. If I don't, I don't.

Sometimes, too, I just have to talk to myself and give a reminder that people like neighbors who might stop in just mean to be friendly and they're not trying to be threatening. Looking at it that way sometimes helps.