I think that you have a good heart. It is very kind for you to consider his feelings.
I think sometimes when we are desperate for friendship we sometimes take what we can get.. Do you feel that was the case for you? Is he really someone you would be friends with if you were not lonely at the time?
Do you really click with him? Or are you just his friend out of sympathy?
I dont think its fair to him, to pretend to a friendship you are not really into. If you do want to be his friend and enjoy his company and conversations when he is not being weird. I would def. keep him around.
I would then tell him, lightly.. that sometimes he makes you uncomfortable. Maybe explain to him that some of his behavior has made people uncomfortable.

I hope it all works out, Im kind of tired IDK if any of what I said even makes sense right now.
Good luck