Personally I do think that what people think of you matters. It's just that we tend to take and give the advice of "don't care what anyone else thinks" too literally. To me it's not that it doesn't matter what others think, because it does, it's about being able to discern between the opinions that matter and those that don't.

For ex, let's say you have person A and person B and each has an opinion of you. Person A has known you for years, is a relatively non prejudiced, open minded, empathetic, well meaning person. Person B has known you for less time or barely knows you, is more conservative and prejudiced and there's reason to suspect they don't care that much about your well being. Whose opinion is of more empirical value, more likely to be based in an unprejudiced, well informed judgement and assessment of you?

That's how I see it. If well meaning people who are relatively unbiased by false societal norms(like sexist, homophobic, binary beliefs) have an opinion of you that is also relatively well informed through some degree of proximity and knowledge of your situation(for ex of what depression and other mental illness is), we should consider what they have to say, it does matter, they might have a point. On the other hand, if it's some random bully, some prejudiced person in a prejudiced society, if it's someone who doesn't know us at all really or never tried to, their opinion may be valid for them but it's not based on much and it's too heavily biased by external factors, stereotyping and so on.

So basically I think that there's some truth in the saying but it should be more like saying one should learn to not take every opinion, good or bad, to heart and first consider how well informed and unbiased that opinion is.

It's even a bit dangerous to lead your life without caring what others think. First of all, whether you emotionally care what others think or not, unless you are financially completely self sufficient without the need to interact with anyone else as part of your work or business, how others perceive you does make a difference and it will make a difference and often it can be severely biased and unfair whether you care or not. Secondly, I do think we need feedback from others, to calibrate our own behavior and way of thinking cause we can't see ourselves from the outside and our own emotions make us biased in a way that can alter the way we perceive ourselves and our interaction. Like, for ex, someone who is socially anxious can feel like they're ignored or not liked in any social situation and that is often exaggerated. Or you can just not realize that you've been hurtful or wrong about something or some interaction. Feedback is good.

Another example is ho you shouldn't "care" what sexists, homophobes, racists or xenophobes think, you're not less of a person, you're not wrong or ugly or whatever and you shouldn't hide or feel any less human and valid. That's very true but at the same time we can't ignore the fact that people outside the norms, whether they care or not, face negative consequences, more or less, depending on the local culture. So, someone can be out and proud and still get denied a job for being gay or trans, or even worse they can be physically assaulted and ostracized in ways they can't control and that do affect their lives in quantifiable ways. Of course it matters how people perceive you.

In my opinion, by saying that it should absolutely not matter what others think, we're also taking the blame away from those who discriminate and act in an insensitive way. It's like victim blaming because freedom of expression doesn't mean freedom to hurt other people, it doesn't mean we should be free to discriminate and hurt others because that actually takes freedom away from people subjected to that discrimination.

In fact, generally speaking, what people think of other people, of places, foods, objects, etc does matter and it serves a purpose because we as humans learn and evolve by learning from each other, by taking in information others have uncovered and building on it, we adapt by example and we make better things to improve our quality of life by observing people's needs and wants and how things affect them.