I totally see where youre coming from and also because you do hear a lot about people with intellectual disabilities being taken advantage of in those kinds of contexts. But from what youve said it sounds like nothing bad at all. If you knew they were disabled, or in some way suspected they would not be able to easily refuse things, and dated them because you wanted to take advantage of that, that would be wrong, but that wasnt the case. You both wanted to date, so you dated

People with disabilities are often thought of as inherently asexual/aromantic, and sometimes the opposite as being hypersexual, but that's not true. As for thinking of them as a child what that really is coming back to is consent issues. You might want to look into information on how people with intellectual disabilities can have safe relationships to help with your anxiety about this. And also because its always good to learn new things, right? "disability and sexuality" is a good search phrase to find info on the topic.

STuff like intellectual disabilities and related conditions come in a variety as wide as the number of people on this planet, sometimes its easily identifiable and sometimes its not. If you have ever told someone about one of your conditions and they were shocked maybe you can relate that. On that note its worth mentioning he may not even have an intellectual disability since that was hearsay

It would help to talk to a psychologist about this i think, its kind of a terrible irony that the thigns we're too ashamed to talk about are often the things we need to talk about haha.