I can really relate to where you're coming from. The very few people with whom I have confided in about my anti-sexuality have all immediately inquired if I had experienced some kind of traumatic sexual abuse. This trend continued with the two therapists I saw, they also even asked if I wanted to try and explore repressed memories just in case.
It doesn't make me angry exactly, but it does get a bit irritating when people can't seem to fathom that you can simply be who you are without having undergone some kind of trauma that requires "fixing."