If he was literally on your property, ask him what he was doing. You have the right to know. This sounds creepy.