From what you’ve stated it seems you’ve already crossed the line with this man. And that he’s pursued you. I’d tell him you can’t pursue or be with him while he’s still married. Deal with the pain of not speaking with him and move on. This too shall pass. It’s the right thing to do. Trust me, you don’t want to be the reason his marriage breaks up. And you didn’t answer whether they have kids.