Do not worry about your husband and what he is feeling, that is his job. He does not get any claim on telling you what to say, think, feel, do. Forgiveness is for you, when you don't need to "consider" the affair anymore, don't need to let it affect you and your actions, after you have healed. Were you to hold it "against" him, not forgive him, that is robbing you of your energy, not hurting/getting even or causing him any difficulty, just you. You would be the one that would be holding on to the dirty laundry.