I believe the right thing to do is to regularly bring your eyes back to meet theirs. This helps show that you're engaged, but it doesn't make them feel like you're staring bullets into them. When not looking right at them, let your eyes slide off to the side rather than staring down at the floor. This can give you the 'engaged but in thought' look.