Driving is like a lot of skills,it gets much easier with practice. So practice as much as you can. Tell your instructor "I am an individual. Do you have a helpful suggestion? You can do your studies after my lesson."

For example, if I had a student who hesitated to put their foot on the pedal, I'd have them watch a video with driving in it, and have them put their foot on the gas pedal--a soft ball, book or pillow on the floor-- when they thought it was appropriate; then when they could see that the driver was stopping or turning, to put their foot on the brake pedal (another soft ball or pillow or book or what have you--just something under your foot); to get used to connecting their eyes to what their feet are doing.

Don't let this guy distract you and upset you. I doubt he really means to do that, but since he is, you need to tell him and ask for helpful suggestions to overcome your particular problems as you are learning to drive. After 40 years he should have some; if he doesn't he isn't a very good instructor.

You'll get better at driving. I have faith you can do it. Courage!