What if you start by telling him that the second trimester is the most stable part of pregnancy? You are generally past the miscarriage risk, but not yet where you can be at risk of premature labor. What if you thank him for the concern, reassure him, and tell him that in the future he can resume worrying, but for now it is ok to take a break?

I agree with Divine that therapy for him would be fruitful now, not even so much for your sake (that, too, of course), but for the sake of the baby. Imagine your H growing into an overprotectice, excessively anxious helicopter parent - that might feel suffocating to the child who will one day appreciate an opportunity to spread his or hew wings.