It might be that you're not picking up on others being passive aggressive. That's probably a hard one to pick up on if you do have aspergers. They act overtly fine but give subtle hints that they're bored or mad even if they say they aren't mad or bored.