I've had this experience so many times. From kids at school, from coworkers, from my step-dad and members of his family, from church people. Some of them just wanted to hurt me, some thought they were 'helping'. It's tough to deal with even when you do your best to cut them out of your life, which you should do if possible.

Remember that no one gets to define you, except you. The only labels that truly matter are the ones you give to yourself. Don't help cruel people harm you, use those claws for protection.