I think there's a difference between driving past someone's house to get somewhere else (like the grocery store, a job, whatever). Because you don't notice that house. Its's just scenery.
But when you drive past the house with the house as where you want to see that house, that's different. Because objectives are different.