I think this happens to people regardless of whether or not they are on the spectrum. People rarely listen to me too and it feels like nobody cares what I have to say.

What others said about either avoiding these individuals or being more firm in your words is the best advice I can think of.