The idea of "having some fun" would have me concerned too, but I don't think I would divorce a man after one incident. Now, if he has actually had an affair or even a one night stand, I think that would be a different story.

My Dad lost his wedding ring a long time ago and never wears it. But I don't think he's doing it to "have fun"