You shouldn't need to tell him "why" you want another Doctor. That's crazy.

Since you have too, you should just say: "I am no longer satisfied with the level of care that you provide, and I would like to explore other options." Just leave it at that. Don't go into detail. Set your boundaries and stick with them. Write what you are going to say on a note card, and practice it, even take it to the appointment if you need too. He will try to weasel the details out of you, since that's what he does for a living, but be firm. Also be aware of your body language. Sit leaning back on the chair with you're right leg crossed over your left, don't slouch (since that is perceived as being unsure), make sure your face is relaxed, make direct eye contact, look somewhat serious, but not happy, kind of like a blank stare, rest your hands in your lap, but don't clench or start wringing them, whatever you do, keep your hands away from your face/hair, especially don't cover your mouth while talking.

Did you get all that? lol.....slightly manic today apparently.

I've been reading this book "How to Say it At Work" by Jack Griffen, and there's a whole chapter on body language.