Yes, that's agoraphobia, and very familiar.

The Dr. Weekes method I adopted was to just go, and take as much time as you need, but don't abort the mission. So in a situation like you are facing, leave early - if you have to pull over and take a breather in a parking lot or somewhere along the way, do that, but then keep going once you regather yourself. No matter how slowly you have to go, or how long it takes, complete what you need to do. That's CBT. Once you do it, your brain will begin to rewire itself to become calmer and more relaxed as you do these kinds of things. It will take a while, and much practice, but it generally works if you keep doing it enough.

I have memories of needing to go to a craft store, to get supplies for a DD school project. By the time I got there, I was not doing well, but rather than run home like I wanted to, I just sat in the car for fifteen minutes until the panic began to settle down- then shopped, then drove home. Over the long run, doing this does work. Your brain begins to see fewer threats. I'm still the person who shows up for appointments 15-30 minutes early, for time to sit in the car and acclimate if necessary, before going about whatever business is at hand.