This stood out to me the most because you're contradicting yourself, here.

If you know your partner has had years worth of struggles in which you sympathize with, then having an affair on him on TOP of it all, is not a "nice" thing to do to him. Neither is staying in a relationship with someone you don't want to be with, simply because "you're" struggling to end it.

I'm not getting the impression that you care about him so much that you can't fathom breaking his heart with a separation if you're not at all concerned about how this affair will affect him.

I'd question why you're still hanging on. What are you really afraid of?