It is NOT okay for a doctor to yell at you ever. You should switch if for no other reason than you are afraid of him. I respect care providers that call me on my crap but bullying mental patients is something else altogether and I have seen it happen.

It really sounds like on some level you are asking him for his permission and you don't need it. Especially if you have an alternate care system in place already. Frankly the most you owe him is a phone call saying his assistance is no longer required.

You are making a change, it no longer matters what he thinks or how he feels about it. The only thing that matter is what you think and how you feel. Bringing along support was a great decision, it sounds to me like you are making good decisions for yourself. Don't ask questions, just make statements.