I just wanted to say that I can relate a bit to what you are saying. You don't sound strange to me at all.

I have similar things happen to me. I have a hard time with anyone sitting behind me or coming into my "personal space"-- which is rather extended. I tend to have a physiological response in which I get the whooshing sound in my ears, rapid heart-beat and general feeling of panic.

Maybe this is the case for you? Might be something you would want to see someone about?