I think background does make a big difference. If you flirted or dated and then complain about him, he can always bring it up that you flirted with him and he thought you were encouraging him. If he is just a coworker (which he isn’t), then it would be a different story.