Probably based on the idea that the person is usually there on most occasions they are noticing the absence. does not necessarily mean that they enjoy their presence.

If you're part of a group of say 5 people. One of those people annoys you but your typical group is always the same people, you would still notice the one missing.