I know it's not entirely the same, but I have always been uncomfortable with my body, from being raised by a narcissistic mother who constantly criticized me but then also dint help when it came to food and such. This last year, I realized that I am much more comfortable wearing men's boxer briefs, even tho I am female and identify as female. I've had people ask why, and my response is usually along the lines of "why do you care" or "it's not hurting you so eff off"...

So my answer, you do what you need to do to feel comfortable in your own skin, regardless of what they think. Often, they would never understand anyway.