#1 thing is don't pressure him or make him feel bad about himself sexually. Try to seduce him and tell him what he does that you enjoy, but never criticize or seem disappointed. Try initiating intimacy without sex several times for each time you try to have sex. If he has ED then sex is stressful for him when he loses it, so if every time you go for hugs and kisses he has to perform that will end up making him want to hug and kiss you less to avoid possible humiliation. Vice versa if he is relaxed making out with you not worried about sex and penetration, he'll be more likely to be in a better place for sex. Most of the time it's mental, like performance anxiety I guess. If he's older might be worth a trip to the doctor, since there's also loads of medial things that could cause it. It's very common, just not discussed much. So in a nut shell, having been there and battling out, freaking out on him will only worsen the problem so you have to think smart and play smart now. If you can, and he loses his erection during sex, keep going by yourself while kissing him passionately. Guys are really turned on when women orgasm while kissing them.

** I just read the part where he has a history of affairs. I would say make sure he's not having one now. The best way to know is to literally trust your gut feeling. My gut has been right so far 100% of the time, I've been highly suspicious of every cheater from early on.