Wow. You guys are so judgemental.

The only problem I see with this is that it seems like with the way he told you about this that he's unhappy with your sexual performance. It also sounds like if he had the chance he'd have an affair with her behind your back.

If you were to leave him, I don't think this should be why. After all, him being consentually attracted to someone doesn't change who he is as a person.