I have to be careful how I say this, because I'm speaking as the guilty one in an affair. My wife certainly did things wrong - she's lazy, irresponsible, physically violent when she gets angry... But you know what? What I did was my fault, NOT HERS. Ok, so I was on a manic episode and made poor decisions - I still made the decisions. My bipolar doesn't excuse my deliberately seeking out and reconnecting with an old flame and leaving my wife for her.

Your husband is not behaving like a person who regrets his behavior, just one mildly annoyed with the consequences. If he was sincere, you would hear genuine apologies and see dramatic changes in his behavior. If you think your time together justifies an attempt at reconciliation, lay all this in the open for him, preferably with a therapist or counselor. If he refuses, you have your answer. Cut your losses and get away from him.