This is a tale as old as time. There has always been "that guy" and then there's always been the one really nice person stuck in the middle. I don't think it ever gets any easier...

I think you are doing the right thing and being a good person. It's hard to say which came first; his behavior or the exclusion from groups of people (which in turn could have caused the behavior). He could be depressed, as one of the symptoms of depression is often irritability.

Maybe you could try talking to him about his inappropriate remarks? Next time he says something like that, just say "hey, that's totally inappropriate, and I'd appreciate it if you didn't say that." If you keep doing that, maybe he'll start to get the hint. He probably just needs more experience socializing with people (which is a double edged sword... He needs to be socialized so people want to be socialize with him, but how do you help him to socialize when no one wants to be around him?).

Go and try to have a good time. Don't worry about what anyone else is thinking. As long as he's not hurting anyone, I think it's fine. If he starts to get to you, let everything go in one ear and out the other. Do try to have fun!