That's sounds like pretty abusive behavior on the part of your doctor.
I went through something similar at the end of 2019 and I had to report it to the hospital/clinic and to the state mental health board. It is not easy to do but somthing that the person at the state agency I reported it to made me feel like it was definitely the right thing to do. He told me that the behavior was of a significant nature, but even if it ended up that nothing could be done for my case as that sometimes happens, that the records of my complaint would help protect anyone else in the future who may need to file a complaint about the same provider. He said if multiple complaints are filed then that gives their department power to act.
I don't know if the system works the same where you live but it's something that may help you decide what to do.
If you have not already you may want to request your full records from that doctor if that is allowed where you live. I requested mine from the abusive provider and found that he had made claims about things that he had no knowledge of, and he made very wild sounding accusations, that in my opinion could only serve to prove he acted in an inappropriate and abusive way.
Hope you can start feeling better from this. It's been hard for me and I'm still working through the feeling of betrayal, but I refuse to give up.