Don't down play what is going on. This happened with my wife and our coworker. It started out with her just saying it was harmless flirting and then it became her confiding in each other before becoming a full blown out emotional affair. He was going thru an ugly breakup with a gf of 8 years and apparently she had been lying about being happy in our relationship for years and they were each other confidants. Before long they were telling each other they loved each other and that she wanted to leave me for him and so much more. I wanted a divorce because I was done but she agreed to stop seeing him or talking to him (and it helped a lot that he got fired shortly afterwards). To be honest, I think an emotional affair is worse than if they actually slept together. I told her if she messes up one more time I'm done and filing the paperwork end of story. It's been a bumpy road and we're still working things out (this just happened in Feb of this year) and it seems like she's being faithful and honest.

The best advise I can give you is to go with your gut. If you know something isn't right then more than likely it isn't. You should come first and never second in your own marriage and if that is the case you need to put your foot down or end it. If you don't you'll just drive yourself crazy and it won't end well for your mental health. I wish you the best.