Oh boy, that's a tough one. Good for you for fighting to stay in the relationship and figure this out instead of just going along with your anxious feelings.

You have lots of nice things to say about him, so that part sounds like you don't "need" to give up the relationship. Unless there are bad things that you don't really let yourself see. (I'm thinking of things like when people have an intuition that their partner is controlling or cheating, but they let love blind them to it.)

But do you want to leave? When you are not feeling anxiety, do you still feel this way? Maybe your anxiety is a way to protect yourself from getting close to him. Or maybe you feel generally anxious and it feels better to kind of focus your energy on the nearest human target? Do any of those things sound right? Are there other emotions that come up besides the anxiety that leads to panic?

I have the borderline trait of switching between thinking my husband is an angel and then thinking he just doesn't do it for me. It's really frustrating to always be assessing "is he the One" year after year, but it's worth it to work through my own doubts and be with him. I hope you're able to figure out the right answer for yourself. Sending good thoughts your way.