There’s a saying that what other people think of (general) you isn’t any of your business. You can be the most upstanding, kind member of a community and there could still be someone there who held irrational hatred for you.
Could this be a fear of rejection on some level, do you think? I have had similar feelings before, mostly because there have been times I’ve felt unsafe around certain people I didn’t know. It’s not that I wanted them to think well of me, I just didn’t want to feel threatened or “pushed out of” a place I had the same right to be in as them.
There may be a different underlying reason for you, but it comes from somewhere and it’s only you that can answer why you feel that way.