I think the therapist should be told and there could be a conversation about boundaries.

I once had a therapist who had his office in his home. The way things were set up I had to ring the front doorbell, and then walk through his living room to get to his office. After the session I exited out of a back door in his office.I found this arrangement disturbing. My appointment was late afternoon. There would be cooking smells, and sounds of children playing. I think I should have brought up how it disturbed me walking through his house. I would never do therapy now under such conditions. I would find a therapist who had a separate office. In this case, seeing the therapist's home (walking through it!) did nothing to enhance therapy. I don't remember making very good progress with him, and it ended when I moved to another state. Looking back, I think I should have said something. I wish I would have told him that I felt more comfortable using his outside office door to both arrive and depart. It isn't the same as this situation, but now I think it would be best to bring up driving by the house. The therapist should be more than equipped to handle talking about this situation.