First of all, stop blaming yourself.

No matter what you did in the past, he choose to remain with you and two wrongs do not make a right. If he never truly forgave you he should have spoken out about it, same way as if he doesn't feel sexually pleased with you or whatever the reason is he should be able to communicate with you especially after counseling and rebuilding trust. Ask yourself if you're willing to forgive him or if there is even hope for the relationship to gain trust again, you also have a choice to stay with him or not but don't feel like you have to just because he did with you. It's going to take a lot to establish a trust worthy relationship now that you've both cheated on each other before, sit down with him communicate and make a decision from there on.

Best of luck to you!