I do not know what state you are in, and family law is by state - it is not federal. I doubt that the court would make the custody arrangement dependent on whether the guy was with you - it would be weird because the affair was private and discreet and did not involve the children or involve cohabitation while married or anything of that sort... it is not that the guy was having drunken orgies in the presence of the minors, right? But it varies, very much depends on the state, and sometimes depends on the circumstances of an individual case and predilections of individual judges.

To sum up: before making wild assumptions that derive from your ideas and possible misconceptions about the law, you should have sought legal advice.

"I started to worry he might lose custody of his kids if we were found out. I couldn't live with that." - all of that worry was unnecessary and instead of worrying and making assumptions and making ultimatums based on assumptions that can be wrong, you or he or both of you should have sought competent legal advice.

He also needs a professional mediator and possibly a counselor who would mediate and guide him and his wife through the process of their (hopefully amicable) divorce. He is clearly not the first or the last guy on this planet who is "afraid of telling his kids about the lie he and their mother have been living for years now". It is a standard, typical issue and professionals must have developed standard, tried and true methods of dealing it with it. No need to reinvent the wheel. I am unsure he even needs to tell the kids about the "lie". He and the wife need to be coached in how to present the situation to the children.

So, to the extent that his feelings for you are authentic and his desire to divorce his wife is true and he is just blocked by the fear of confrontation, the whole situation was due to multiple failures to engage professional help when necessary.

Alternatively, he used the fear of confrontation as an excuse to mask his lack of a true desire to divorce the wife.

You know him better so you can tell which is which.