It's really no big deal, I know it feels like a big deal but just the fact that he seemed normal at first should be enough to dissolve your feelings of shame or guilt. You simply didn't know. Intellectually disabled might not mean retarded, it might mean he has autism or perhaps even something as simple as tourettes. And you only hung out with him for five days you said? This is a very short period of time. It's not like you were hanging out with him for a month and still didn't notice anything. No, it's not like being in a relationship with a child, he is a grown man so you shouldn't feel guilty about that.
Just let it go and chalk it up as experience...not a big deal, at all. Just forget about it, you will forget about it when you start dating someone else anyway.