It shouldn't be a problem as I know plenty women who love wearing boxers, and nobody judges them for it. What's good for the goose is good for the gander. Gender equality and all that...