I would never admit to this in a bajillion years if it weren't for internet anonymity, but I've done the cross-dressing thing once or twice. Although never outside the privacy of my own room. It's always after a period of time when I've tried to go cold-turkey on masturbation, so I get really, I guess the term is "pent-up." That always leads me to some weird binge in the end.
I also experience that same turn-around you describe, although to a bit more of an extreme degree. I've physically hurt myself more than once in the past because I felt so terrible and ashamed over what I'd allowed myself to do.
So I wouldn't say you're alone. There's likely a lot of people who experiment this way a least a little who aren't too keen on talking about it, even in the anonymous environment of the internet.