Speaking as the wife of a man who had an affair (he didn't remember it due to his illness, but it still happened). I think you should end it. Do you really want to be the "other woman"..."homewrecker"...etc?

His behavior is wrong on so many levels. While I think seeing his reaction with you so close to his wife would be interesting...why play games? He's already playing them with you...aren't you better than that? Don't sink to his level. He's obviously an accomplished liar. What reaction would you want? Panic? Fear? What if he acts like nothing is wrong?

My advice. Run, don't walk away from this man.