I think there may be a distinct difference in celibacy and asexuality. Celibacy does seem to imply choice in my opinion, e.g. a Catholic priests take vows of celibacy. I'm not that familiar with asexuality but it is defined by others as lacking interest in sex.
That still doesn't explain why they get mad at you. People get angry over sexual and gender terms. I don't know why.