My wife usually worked around all men and really prefers it. She hates working in offices full of women because of just the thing you're describing. I don't understand why it happens that way, but there seems to be some kind of weird power struggle that starts to occur.

Wish I had some advice to offer, but I've seen it myself.