In much of the country, masculine is normative, "human", while feminine is, well, feminine. Thus, often someone wearing traditionally masculine clothes and colors is taken to be simply "human", not gendered; while someone wearing feminine clothes is taken to be feminine, gendered.

This way of thinking, seemingly common in the wider world, might be infecting or overwhelming the world of which you speak.