I would suggest some therapy to help him with his fears. There are therapists who list working with learning disabilities and disabilities in their areas of interest and expertise. I think social anxiety (fear) might be something they see often and could help him with. That really seems to me to be the place to start to help him broaden his social life, including friends and work. Also they might know of other great resources available to help him meet people and find employment.

Good for you for realizing that while it's so nice to have him with you, it isn't in his best interests. You're a good mom!