It's pretty normal, I believe, like it or not. I would occasionally find my mind wandering even when I was with my ex. I would've never, ever have been unfaithful to her (she should have been a fraction as loyal...), but the mind thinks what it will. So, there I think you're fine.

Not fantasizing about your boyfriend at all, however, worries me slightly...namely that coupled with the issue you mentioned in your "Black and white" topic. Are you attracted to him physically?

That being said, I do recall how you mentioned your coworker behaves towards you. Personally, knowing that, I don't care if he (insert every physical trait you're a fan of here). He's sort of a (your word here), IMO, but that's me.