Babies, once their eyes actually see STG- in the beginning they do stare blankly which many mothers decode as a deep attachment- will follow their mothers eye movements, trying to see what she's looking at cause that's important.
Prolonged staring is a sign of threat most ppl are uncomfortable with- we do share this with many other species. When you look into other ppl eyes you are actually checking what their intentions are.