Be direct. Men appreciate it.

Just text him and say that you don't want to go on any more dates and that you'd appreciate it if he didn't call into work especially to see you as it makes you uncomfortable (you can be as blunt or nice about how you word it but unless you make those 2 things clear, he may read other things into it and think there's still hope).

If after that he still comes into your work just to see you or still tries to get another date then he's entering stalker mode but I don't think that'll happen from what you've said. He'll probably just be slightly embarrassed and you'll never see him again.

Being rejected is one of the pitfalls of going out with someone and most men would prefer a direct rejection quickly rather than having to endure a state of misplaced hope only to find out later that there was never a chance and that they've been behaving like a ****.