Maybe I'm just a weirdo, but I've never felt controlled by the sign on the door before! Although obviously I see why people would and normally should be using the bathrooms for what they are/identify as. I'm pretty feminine but I have used men's bathrooms whenever convenient, especially at my last job or in stores/places tailored more for women so men will be less likely to be in there. They've never said anything if they were in there. Just stared in wonder.