Uh.. take you off your meds? Why? o.O
I don't even see my pdoc anymore, I see my physician every 1-2 months or so to report how my meds are working for me, get new subscription, and various paperwork.

But yeah, like the others said - if you feel anxious about it then tell him. Alternatively you could have your physician handle your medications and see a regular psychologist instead, and have your physician and the psych. work together.