I think in such a case it would be appropriate to ask them if everything is ok. Saying something along the line of "I noticed you've stopped by me a couple of times, is this something that I need to concern myself about?"