Personally, I don't see the big deal in driving by, as long as you don't make it habit* or ever make your presence known.

Sometimes the "not knowing" makes something a bigger deal than it is.



*I sometimes walk past my therapist's house. I don't try to peek through the windows or anything and I'm mindful about staying away when I could bump into her. But I walk all over town. Her street is not off-limits to me just because she lives there, and she knows this. Sometimes she'll ask if I've been by lately (usually when she's talking about her garden), and I will tell her if I have. So far she doesn't seem bothered that I know where she lives. I assume that she wouldn't have told me her address if she didn't trust me.