If you have a pituitary tumour is it under control with medication? Once you get that under control you can reduce your levels of prolactin which can boost fertility without the need of clomid. That's an aside though as I think you are choosing the right path.

If he yells at you, how will he feel at 2am when the baby has been screaming for 2 hours? If he is highly reactive don't doubt that anything could happen.

You are making the right choice.