Actually, a lot of people don't care to invest their time helping others or spending time listening to others talk about their problems. It doesn't mean they don't have empathy either, sometimes they are dealing with their own challenges and they just don't feel comfortable focusing on the challenges of others because it can end up with them being reminded of their own challenges that they have not resolved themselves.

This is why a lot of people keep their distance. Some individuals who struggle can be emotional vampires too, that can be very draining. You just need to figure out what you are comfortable giving and not feel bad if you don't have the patience.