I believe self-esteem may be partly inherent but more or less overwhelmingly influenced by external factors. I can completely understand that poor treatment and reactions from people around you either cause or exacerbate pre-existing low self-esteem. And simultaneously, a horrible self-esteem can also give others the impression you do not value yourself and hence they will also value you less. It seems to be an unrelenting cycle.

Of course if you were exceedingly attractive and garnered lots of positive attention your entire life you would be more likely to have higher self-esteem, which subsequently would encourage others to treat you well too. In light of this, I find I respect those who may not be particularly or conventionally attractive, and yet they still have the personality and confidence of one you would think would not suit them - and they become very attractive. Even completely unattractive people can be very well liked and given positive feedback, but I suppose they do not get that 'free ride' that very attractive people will get.

Personally, I definitely believe my low self-esteem has stemmed from being treated poorly when I was younger, as I was kind of.. unkempt and ugly. One thing I did was lose even just a bit of weight a couple of years ago and not only did it make me more confident but others treated me better; it may be a due to either my own heightened confidence, improved looks, or a combination of both, but maybe it doesn't really matter. There's no reason not to try to improve your outward presentation with simple things however, and see if that makes any difference (it likely will).