I don't think there's an easy answer here one way or another. It's going to take time & some deep introspection to figure out how you want to handle this. Sadly, I'm afraid the cold, hard reality here may be that someone is going to end up hurt in this situation. The question may be, who is it going to be, & how can the pain be minimized.

I know you wrote the two of you are participating in couples counseling. And that's certainly a good thing. But perhaps you should consider seeing a therapist for individual counseling / therapy yourself... someone with whom you can, over a period of time, delve into & come to terms with these difficult issues. Realistically, at least from my perspective, this is simply too complex a situation for much of any useful insight to come from us here on PC. I wish you well...