hmmm well it seems that you are being kind. And i support that. People are all diferent, some of us arent great socialisers, but it doesnt mean we arent great people.
I would have Invited him. Its not nice excluding people.
If he was nasty to people then it would be different but from what u say he sounds decent enoUgh just not a very good people person????
Anyway i think you did the right thing.