Eye contact is a form of communication, lets the other person know you are interested and listening to what they are saying.

I know I tease my husband, sometimes when we kiss he is looking over my shoulder! I ask him what that woman looks like or what's going on back there, if it's anything I need to know about

Generally, we look where our attention is; look at the floor and you could be thinking (but the other person can't have any clues whether it is about what they are saying or something else) or you could be miles away from the conversation, prompted by the rug color or pattern or other thoughts, etc.