This happens frequently in work places. I advise you to either tell him directly that you are not comfortable with his behaviour and would he please back off. If that dosen't work I would tell my supervisor and have him/her talk to his supervisor. If either on of those dosen't work or, he dosen't stop, then I would go to HR and file a complaint.

Good Luck,

Eric