I agree that you don't have to answer your door. If I were you, I wouldn't. I never answer my door unless I am expecting someone. Especially at 8:30 at night.

I guess he was trying to be nice but you don't have to accept "nice" just because someone is offering it.

With regard to saying that your door is open: Well, you can undo what you said. Not by taking it back literally, but by not being available to him in the future. If you just are consistently not available then he will come to conclude that you were being polite, you have less time than you thought, he always catches you at a bad time, etc. It is okay to say "I have to get going!"