I think now is the time to start threatening, I would insist on having the drs name and address tell him you will get the Drs number yourself.
If he wont comply then tell him you will be phoning social services on him.
Adults have to stand up for children, they cant do it themselves.
I know this is scary but in my head this needs to be done.