You are not at all useless to society! Your home is your home and not society's home. There is no reason to feel guilty about not answering the door. You simply didn't feel like it. And of course you wouldn't feel like opening the door to someone you think is unfriendly. It is a small leap from unfriendly to untrustworthy, threatening, unsafe. So if that is where your mind went, then it was a protective action to not open the door to him because you don't want someone like "that" in your home.

Might be fun to be curious about it, too. Perhaps he was feeling guilty about not waving back and wanted to say hello?

Maybe another time, when you see him outside, you could mention that you were unable to come to the door the other day (no reason why is needed) when he was there. Let him explain then why he was there. This might settle your nerves and guilt. Not knowing is harder than knowing.

I consider a knock at my door, a ringing telephone, and an invitation to be things I can accept or decline without reason. I don't consider them an obligation, so no reason for declining is necessary. (No reason for acceptance is ever sought!).

I hope this person is a safe person, and that you can find neutral ground to interact so you can make that determination, and feel better.