I've the same issue. I worked as a cashier from the age of 17 to just a month ago and my poor math-skills could be quite a hindrance. Sometimes people will change their minds about how they wish to pay after I've already gotten their change, for example. "Here, you give me this and I'll give you that. Now if you do this, and that's that." I can't do math under the pressure of another's judging gaze. I shut down, as you say.
It's funny, they always sort of group mathematics and sciences together. It makes sense, I suppose, to assume a person well-versed in physics or chemistry would also have profound knowledge in mathematics. That isn't always the case, as you've proven. In fifth grade, I was in an advanced science class, despite being the only student who didn't know her multiplication-tables. I understand how to work formulae and equations perfectly; it just takes a bit to complete them when you can hardly multiply. You understand, surely.
I honestly think our mathematics skills would be much better developed if a rude teacher hadn't instilled into us the idea we were stupid at such an early age.