If he would speak to you when no one else would then he is a true friend that you should keep in contact with. Not many people would do that. It sounds like he isn't socially acceptable and not a people person, but it doesn't mean he won't ever be. It's good that you invited him, you should have. If the friends you hang out with don't want to be there because he is there then maybe you and him can just do something alone, because that isn't right. They aren't real friends if they won't go just because of him. It doesn't sound like there is anything that's so terrible about him.

As long as he treats you right then you shouldn't care what anyone else thinks, because he's a good friend. As far as him walking you to your car and stuff perhaps he just doesn't like the idea of you walking to your car alone, or he has a crush on you.

You did the right thing so whatever happens be proud of yourself. I don't think he'll want to go anyways because he doesn't seem like he would fit in socially.

Good luck to you.