LT:

I have no way of knowing if this is true for you, but it's something I'd be feeling, so I'm going to throw it out there in case it resonates with you at all, and if not feel free to totally disregard it.
My twin brother is on the spectrum. Because there's already a lot of stigma and misconceptions out there, I struggle to talk about it with people who don't get it. I worry about reinforcing their biases. I feel like I have to defend his behavior. I got so used to having to stand to for him and defend him with people who didn't understand that it's hard for me to acknowledge the ways in which it made things more difficult for me or to hold him accountable for his abusive behavior. I feel like I'm supposed to be more empathetic and supportive. It feels like people who don't have those experiences don't have the "right" to judge it in the way that people who "get it" do, and so I'm going to talk to them about it differently and interpret their opinions differently. Someone who "gets it" is "allowed" to say things that others aren't.
It's the same with ethnicity/cultural background. I feel a lot more defensive talking about how my father's ethic/cultural background and upbringing contributed to his worldview abusive behavior if I'm talking to a white person from a "Western" background."

If I'm talking to someone who "gets it" I don't feel as defensive.
If I later found out that the person might not "get it" I'd feel betrayed. I'd be upset and feel that I'd been misled into, for lack of a better word, thinking an outsider was an insider. I'd feel like I'd disclosed things under false pretenses.

Whether or not this applies at all, your feelings that you've described are understandable and the way you've addressed it is admirable. I'm sorry you're having to deal with this situation, and I hope eventually there can be a more satisfying resolution.