One thing you're doing that you may not notice... you're noticing him. And when you notice him... he notices you. Every glance of the eye! Even if you're just checking to make sure he's not noticing you... it may be perceived wrong. I say no eye contact and completely pretend he doesn't exist. I've been in this situation before where my awkward glances at someone (like the janitor) were perceived wrong. Give that a shot and see if it works after a week or so. If anything, it will help you to concentrate on more important things than a creepy janitor.