I think he was confused about how I knew. He asked at one point "did I tell you I was moving?".

Maybe he thinks I drove past his house and saw the sign?

I really don't think he knew. Now he knows. I hate myself.