There comes a time when choosing personal happiness means letting go of being right, especially when you can't be both. You are allowed to want to be happy. If your marriage makes you miserable, you are allowed to not want to be married. I don't think it matters why you're unhappy. I know from personal experience that using another person's infidelity over them makes you feel like you're in control. In reality, the infidelity is in control and it sounds like you may use it as a weapon. I get that you want to hurt him back. Revenge is not happiness.

My father told me before he died that marriage is not a two way street. Its a car on a 10 lane highway moving in both directions merging onto a one lane country road in a constant loop. It takes work by both people to navigate things. Its complicated, messy, and chaotic. Sometimes it works and sometimes it doesn't.