I think you should listen to your wife tbh. Having a fantasy and reality are two different things and many people have fantasies that they wouldn't actually want to act on.

If you push it, you may just make your wife hostile or worried, or both. She's told you how she feels..