Hey Olm
I'm 18 and I know exactly how you feel like... being the outsider, never fitting in, being considered the weird kid, social anxiety.
There's this unwritten rule among the "weird kids" at school that one does not mock the other weirdos for their quirks... at least that's what I've learned. You don't seem to be very enthusiastic about the gaming thing, and they didn't treat you very well either from what you wrote I think...
If I had to give you an advice, don't burn your bridges, try to keep some superficial contact with these three guys, but don't try to talk yourself into seeing them as good friends when they really aren't. It can make you doubt your own perception if you feel you're only surrounded by d*ckheads and insensitve or mocking people, but sometimes, if you're an individualist, it might just take some time to find good friends.
Ways to better self esteem... well of course it would be good if you sought out therapy (go to a therapist, not psychiatrist, the latter just throw pills at you!). Also, you can use the internet and sites like this one to find people who can relate to you and with whom you can talk.
From my own experience, your family and those close to you often don't really get what it means to suffer from mental illness, they either don't take you seriously or devaluate the significance of your suffering. I think this is partly due to a lack of awareness for these issues in society, and partly it's probably you have to have experienced it yourself to know you can be tortured by your own thoughts. So try to not let it hurt you that your mother doesn't get you need help. At the same time it's understandable if you feel angry towards her of course.