It never crossed my mind to be bothered when students told me they drove past my house. Any number of people drive past my house on daily basis. It is public space. Others get to be on it. Now if I lived in the middle of 50 private acres with a fence, it might concern me, but mere drive by peering in a suburban neighborhood is not something to get worked up about in my opinion.