it sounds like you wear the women's yoga clothing because it's comfortable. a lot of garments like that - in the sweats family - are pretty much unisex anyway. so, it's not a big deal. please be aware that yoga sweats and shorts are also available in mens' sizes. an example of a chain that sells both men's and women's yoga clothing is
http://www.lululemon.com/ . all the best to you.