I always wonder how anyone who is working has the time to go out of their way to harrass someone. I'd ignore him too unless he uses the phone and then I'd go to HR.
Good luck. He sounds creepy.