Being with a married person is worse, IMHO. Seeing someone outside the marriage basically showed he wasn’t committed to either of you (and if he was the type to commit, he wouldn’t have cheated on his wife. Sorry).
Have you discussed the future with your current partner? If he can’t give you what you want, what do you want to do about it?