Growing up I was labeled disabled by others outside my family because I was deaf. I didn't feel disabled and I was still expected to graduate, get a job, go to college etc, until I was labeled with a mental illness. then my fater and his side of the family treated me differently although they also denied that MI was real.. Now I have many spinal problems am in constant pain and truly feel disabled because regular day to day stuff is so much harder. The physical stuff multiplies the mental disabilities.
Yes my self worth has been drained out of me. starting with the MI diganois, mostly because I was treated differently and anything I said was disregarded as inconsequential. Now with physical limitations too and being unable to do regular housekeeping my self worth has been obliterated, I can't imagine why anyone would want to be around me.
I have no answer for you about how to increase your self worth it's something I struggle with still.