I do not think that our mental statuses are fixed, like if I were self sufficient I would not be affected by anybody's words ever. I think they are dynamic. And the circumstances also shape us and determine our behavior eventually. I acquired a lot of confidence despite the stigma of disability but when I moved to the US because of language issues I went backwards regarding self confidence. On the other hand, coming back to my country from the US every holiday was a moment to experience a lot of self affirmation: for people in my country if you are employed in the US you are a kind of hero. They believe you are brilliant and feel proud of you. I had to admit that not everything related to confidence came from inside and that I relayed on other people's views and opinions on me more than I wanted. Going back to the point of being in the US, I felt inadequate and out of my league all the time, like when I was a teenager. Then I met several foreigners and could see that I was not the only one. But that did not calm down my anxiety and my feelings of inadequacy.
Observing the world and how people live, I can see that some people get set aside because of certain conditions: poverty, disability, sexual orientation, ethnicity, and sometimes, for some other characteristics that are different from the mainstream ( some characteristics are ver silly such as the type of clothing they use). I am not sure if you have any specific difference that may cause you be vulnerable to discrimination. Or you think you put some barrier for people not to approach you. When you write, you seem insightful and intelligent, very kind. It seems that your confidence issues are profound. Perhaps if early relationships were OK, there may be something else related to the way your brain works. But maybe the bullying was so intense that it was a trauma for you. In any event, almost nothing is written on stone in your life. Circumstances will change and you, too. However, at the present moment you are hurting and that covers your horizon. Please, do not feel you should not complain as your condition is not that bad. The truth is you are suffering a lot. You crave real human contact and it is very painful. I am sending you a big hug and please, hang on in there. We care about you