I used to have that problem all the time until I went deaf. I guess I'm still not looking at their eyes and more their mouth but I use to not even do that. It's still very uncomfortable for me. I use to look down all the time now I have to read lips! I hate it.