Personally, I don't see the issue at all. Your wife just may feel uncomfortable because she (like most people) have been raised to believe that men should wear men's clothes, and women should wear women's clothes. However, you're not hurting anyone by wearing women's panties, they make you feel comfortable, what's the harm?

Have you tried sitting your wife down and having a serious discussion with her about this?