yes it sounds like it is the time to actually tell him he is bothering you and why and see his reaction, and let him know why. if he stays and wants to continue the friendship then you must set boundries.