With all due respect, the man doesn't really seem to be doing anything to you - as in anything you wouldn't allow. All married people get bored once in a while, and when people spend a lot of time at work and not that much with family/friends, it's easy to fall in lust. Again, with all due respect, it just seems that you and him have sexual chemistry and you're trying to be righteous while your hormones are being louder than your reasoning. You don't mention love, passion, strong feelings talking about him, so... Why worry? He clearly means nothing to you, really. Use the lust for fantasy, if you may, but, you know, that's about it. Peace.