I agree with Argarwen, the school Counselor would be the best for this. Or if anything, bring this up to the administration. If you're not having trouble with the coursework (or it seems like he is being too lenient and you feel marginalized/not learning enough?) then his switch in attitude sounds kinda unprofessional. A teacher is to be fair to his students, but also understand when certain needs must be accommodated.

The Counselor might be a good mediator, too. I know my high school one was actually pretty good at making sure teachers and students both understood each other's limitations and needs. But that really is a case by case basis.

And who knows? Maybe Mr. P has just never been really exposed to someone with anxiety and doesn't quite know how to handle it? Sometimes not everyone has the tools and skills to understand sometimes.

Anyway, I hope it all works out! And if it comes to it, just remember, you might only have to deal with him for the school year and after that, hopefully no more.