I do care a lot. I don't think this is a silly thing to worry about. Even people aren't consciently thinking about how they feel about you and what they care the most is themselves. There's a thing called body language, facial expression, the voice tone. I mean most time the words they say don't matter, what matter is what they don't say to you (because they don't want to share it with you) and all the inconscient body language.
In my life I had to relly on this skills to survive and predict how I should behave, and how I still know most of people don't really like me (I'm an aquitence), it continues to matters. When you are insulted a lot, in random circunstances, by people who sometimes are nice to you. I just have to be attemptive. Recognize if I am a burden or not to others, and most times I recognize they will be better without me and they don't really want me there (making me feel invisible). There's no away I wouldn't take seriously the true way they feel about me and about some behaviours and thoughts of mine. People prefer funny and outgoing people.