Self esteem is all you. That is correct and you will need to embrace those aspects of yourself worthy of love. Do you love others? Do you resist negative people and jerks? Do you help others when you can? Do you feel bad when you see suffering? Do you try to make others smile when you can?
There are many things that make us who we are and as trivial as society tries to make them "those of us in the know" know what is important and valued!