I don't know; endo docs are the medical "stress" doctor and it sounds like you were talking about things in his field? Having children IS stressful to your endocrine system (production of hormones) and if you have problems there already, it is probably a good idea to wait until other stresses in your life that you can control, are better?

However, that being so, you may not have liked (duh :-) or misunderstood how he expressed himself; he could be a doctor you don't want to work with because you don't like his personality/manner but I don't see anything technically wrong with his advice.