Ugh how unkind - that sounds like bullying to me. Anyone who uses the word 'retard' in this way is someone to avoid as much as possible.

I would bet not everyone likes him as much as it seems, probably just afraid they will be targeted by him. Hold your head up and ignore him as much as you can.