I believe you did the right thing in clearly expressing that divorce is the consequence for cheating.
Then he walks out on family life with you and your daughter. He was walking out through cheating, but wow, as painful as things are for you, such icing on the cake by him doing that. That has got to be the most immature thing I've read in a long while of a father behaving. Not even a walk outside to cool off. Of course you have a heated argument. Who wouldn't?
He betrayed not only you, but your daughter.
I do believe in exceptions to "the rules" where two parent homes are concerned in raising children. A toxic marriage such as what you wound up in, will consume all of your energy away from this crucial time in yours and your daughter's life.