Being terrified only works against you if you're trying to concentrate. Most likely your full concentration is going to how terrified you are (since this is a high maintenance emotion), which makes you totally miss out on anything they are saying.

I would suggest trying to honestly explain your situation. When you tell someone about it, you'll feel good about getting that off your chest, which will also remove a lot of that anxiety. Try to practice from there.

I too have been in that position and still space out every now and then during conversations (especially when they are unitteresting). Having a lot of practice and being introspective has helped me a lot through the years.

Hope my advice will be somewhat able to help you!