I think the apparent lack of empathy we have comes mainly from deficits in theory of mind. When you have problems imagining what the other person is feeling, it makes it a lot harder to empathize. Lack of interest in people, alexithymia, and difficulties with the interpretation of nonverbal communication also influence one's ability to empathize with others.

I have the ability to feel empathy, but it tends to be delayed. It surfaces when I think about the situation rather then at the time I need to express it. This makes me appear cold and distant. Sometimes, it never comes, because I cannot relate to the experience. This convinced me for years, I was a psychopath.