I've been accused of being gruff and sounding like I'm giving orders to people when I'm not. And, yes, I've been accused of outbursts. In some cases it was true but in others I was simply expressing that I was annoyed or mildly upset. I was not yelling or carrying on, just simply stating my displeasure at something.
My hubby gets accused of being angry all the time by coworkers. He tends to have a straight face and does not emote much so they always think he is angry about something.
Don't know if anything can be done about it.

One more thing.....

Although this is a terrible thing to do, I have noticed that sometimes people who know you have mental health issues or you are seeing a counselor, use that information against you.