Well I have been the partner in the background while the other socialises with his colleagues.

I do feel a line has been crossed here. I would not tolerate my partner doing things alone with their work chum of the opposite sex - ever. It is one thing to have a beer after work with a group from the office another to do so alone. This sort of thing isn't appropriate. Also what is definitely not appropriate is when that work colleague becomes a confidant.