For me, there are 2 reasons why I would care.
A I like the person and want her or him to think sth good about me
B what is being said about me triggers Sth in me

Otherwise, I would not care.

If its A, I'd sincerely think about what that person said and what about it is true (because truth is that usually there is at least some truth to it).