I don't answer the door if I am home alone. I have a phone, if someone wants to talk to me they can call me on the phone. If your neighbor needed something, why could he not call you? And if you didn't answer, he could have left a note, even in your mail box if it was important.

You don't need to ever feel guilty if you don't answer your door, even if it is a neighbor. That is your domaine, you have the right to make it as safe as "you" need it to be, with no guilt.