It won't be easy but as long as you and your wife love each other, stand as a united front and set good boundaries, he should not be able to mess things up

It's a matter of deciding what you both want. It is worth thinking hard about it, even getting counselling if need be.

My concern is having a baby just because of a sense of urgency or desperation. Have a baby because you both genuinely just want a baby and want to love a baby... not because you may not be able to conceive if this one is terminated.