I answered your question from a personal standpoint, IndieVisible.

"I have used both mens and womens bathrooms. And there has never been a problem. No ones brought anything up to me, and that is because I have passing privilege either way. So despite what others were saying earlier, I DO have a privilege within this minority. I pass in ways others cannot."

"It's perfectly legal to go into either bathroom in most states in the US (that I am aware of). This is simply a construct of "sex" segregation. We haven't said rid of all sexed bathrooms. We said that a gender neutral bathroom should be an option so everyone can be comfortable."