Maybe your thoughts are interfering with the other person's voice. I have had times when I am thinking through something in my mind and can't seem to focus to comprehend what the other person is saying. It is usually when I am anxious or stressed.

Also, if there are other voices in the area, it makes it more difficult for me to focus on one person's voice.