This is going to sound brutal and I apologize for the tone, but kick him to the curb.

If he was unhappy, he needed to step up and say what was bothering him so that you both could work it out. Going behind your back and cheating is dishonorable and cowardly and sets a terrible example for your kids as well as a symptom of bigger trust issues.