You are right - what people think of you is important. But sometimes, the anxiety can be deliberating, when you end up walking on eggshells in life just because of the constant thought of what people think of you. It becomes a problem when the thought is constant.

Don't worry about what people think of you. What you need to worry about most is yourself (general sentence, not personal). Worry about your wants and needs first. Then when you're more fulfilled, you can start caring about what people think of you. But in order to care about what people think of you, you need to think about what you want people to think of you, and act accordingly to show them what you want them to think of you