To me, the fact that you cannot speak to him about it is in itself a major concern. Thonk about what sort of marriage it will be when you cannot speak about something that is near and dear to your heart.
It sounds like you would miss having kids with your entire heart. But you would have him. So therefore i ask: What do you love about him?