I can only echo what the others said. Having a condition does not mean you cannot do things in life. You seem well aware and you have plans for what-ifs. You are probably better prepared for this field than average person (many normals go to medical field to get burned out or overwhelmed quickly. Maybe you have advantage here, your own problems gave you much needed resielience).

Your pdoc seems one of the well meaning, but too narrowly focused people. Maybe it is his choice in life, staying safe and within a barriers you created yourself and he projects it into others. He should not be the person who decides about you, as he does not know you as well as you do know yourself.