Typically, the easiest way for me to decide to do something is by thinking, "How will what I say/do affect the other person?"

If I'm going to watch a football game at a buddy's house and I'm wearing my 49ers baseball cap, I leave it one because we'll all have our "gear" on for the game. If I'm visiting someone's grandparents and have my "going outside hat" on, I'll take it off when I go inside.

If I meet someone who is sensitive over their age, I won't us Ma'am because they might feel I'm aging them. On the other hand, someone who doesn't want to be treated that way I'll use Sir.

Again, it all depends on how it affects someone else because ultimately, leaving my hat on or taking it off, using this word instead of that one, doesn't affect me personally at all.