You both need counseling. Individual might be best at first, then marriage counseling. He's hurt by your infidelity, and you're dwelling on childhood experiences.