That's a horrible thing to say!

YOU know that you're not trying to be manipulative, therefore YOU know that their accusation is UNTRUE.

one thing in life is guaranteed, we will be misunderstood and people will have unhelpful opinions about us!

I hope you're can hold on to what you know to be the truth.