Things are rarely what they seem!

In my opinion, I think they have had more of an effect than you may be giving credit for. You cannot underestimate events from the past simply because they may seem insignificant. They might, but you'll never truly know the effect they may be having subconsciously. I really think something is holding you back from giving over "control" to your partner in order for him to help you reach orgasm. You say it feels like you panic when he goes down on you, etc. and you masturbate because it feels safer that way. That is an indicator to me that those experiences you've dismissed are in fact playing a role after all.

Something that may or may not help is to see it as reaching orgasm together, which it is actually. Sometimes I've heard women say their husband/bf gives them an orgasm, or he is not able to give her an orgasm. Seeing it that way may actually block the orgasm because if you are thinking he must give you one, then that reinforces that serious issue with handing over the control, or being totally submissive, totally vulnerable and the fear he may hurt you, which then makes you hit the wall as you say and feel the panic attack feeling.

It should be viewed as reaching it together, so you and him both help you reach orgasm, he doesn't simply give it.

I really do hope things can work out for you, because I can only imagine how much happier it would make your relationship if I consider what you're saying. I can see it's important to you, so I wish you all the best!