I have the same problem too, and my self esteem is shaky but in general I also feel like it is a really powerful thing to look at people directly in the eye and it is distracting, and it makes it hard for me to concentrate!! it can feel really uncomfortable. in this culture it's expected but some cultures don't consider it to be polite.