Hello I.Am.The.End., I'm so sorry to hear you're having a real hard time. I'd like to help you, and it's not because I'm gaining anything out of it but feelings. Some of the things I say might be harsh. If you know it will bring you down, please skip this over. I'd like to be honest. So you would like to connect with others, but you don't connect to groups and you don't feel safe alone with others for fear of being touched?
With groups, you may just not want to deal with groups. If you really do or need to, maybe you should work on dealing with them. You may not like the "group mentality" they have. Or maybe you just don't like the people too well? That's understandable. It may be the people you are hanging around aren't nice, even if you have the same interests as they do. If you don't know how to "jump in" or feel everything you say is taken badly by everyone else, than you may want to work on that. Look at the areas in your parts of the conversation that people don't like, if that's the case, and figure what it is your saying. If it's your mental and physical disability, you may need help from people with the same problems or a therapist. I know you said not many people have the same problem, but there's so many people on this Earth that they may just be hiding. Of course, I don't know what you have, but I'm sure others have what you have. Are you very sure of what you have yet? We are all human, too. I just want to let you know.
Now on touching, I'm assuming you're thinking of being sexually touched, correct? It's understandable if you're only interested in people you are attracted to that it could happen a lot.

You don't have to be alone with people who are sexually available to you. You can make friends outside of people you're interested in or who aren't interested in you. But you said you don't care about people like that? Then maybe it's part of your type of personality. I personally am purposely a little colder in relationships (friendships) with people who I feel might be interested in me, or I think I am interested in. You should set your boundaries, and stick by them. It's hard, but manageable. If you don't like to be touched by anyone, than let others know. I do, and it's never a problem. If you do make friends with people who are sexually available to you, please remember to keep the boundaries you want and don't let anything slip. It sounds like you are very capable of making friends. Maybe you can give nonromantic relationships a try? I've never had a romantic relationship before, and I can say they honestly sound like a lot of work as well...
About friends, you should also ask yourself if you really want friends. Even if you think you don't, remember that no friendship is perfect and takes a lot of work from both sides. Sure, some people may be happy to yap and yap while you listen, but most people don't like to be friends with someone who doesn't talk much. They may start to feel like they're talking to a "doll" and get bored. Plus, some people take advantage of these one-sided relationships. A lot of people are okay with not having friends, or having just a few close friends. I think the later takes up the majority of all people.
Plus, when you talk, how are you sure what you're talking about isn't understood by the other person? It's strange that when you simplify it, they still don't understand? Are you simplifying your words enough? If you are, maybe the other person just doesn't want to learn it. Are you speaking clearly? Some people may just be distracted by other aspects of you, like if you have a strong stare. Plus, some people like hearing about new things they don't understand. Explaining things to people is one of my favorite parts of conversations.
Also, it sounds like you're in school and busy working on what you're doing. If you work too, than yes, you are as busy as a "normal" person. Some people don't understand or remember what it's like to be in school. When they ask for you to do things, please remind them that you're tired or have been working. If you are less available, than of course people won't be as connected to you as before. You're busy, so there's no time to hang around you. Some people just don't want to understand that.
Thank you for explaining that music is stressful for you. It does seem you love it and are willing to stick by it. Do you have other interests or hobbies you can do without other people? There are thousands of things a person can do alone. I personally like writing and drawing. It's a big relief to do something that you can fully concentrate on and have your mind taken off other things. During a very rough time in my life, writing was what kept me going. It lets you be yourself. There's crafts, arts, sports, woodworking, and even travelling that can be done alone.
Often, when I'm in a group and don't talk, people may just glance at me but flat out ignore me. I've always felt like a bad person for no clear reason. Actually, from a young age, most people have excluded me and put me down for no reason. Even as an adult people like to insult me and ignore me. I usually have had friendships with people who don't care who their friends are or people who are mean and use me. There's probably many reasons, but I think the main reason I'm left out often is that I barely ever change myself to make others agree with me. I'm not the type to try to force myself to fit in, and I keep my distinct personality. Some people just want to be around others who are like them, and everyone else who is different is "wrong." Jeez... I've had to learn to deal with different personality types over time. It's not so bad... Spending time alone is not so bad either.