Human sexuality is very complex. As angelicgoldfish05 pointed out, it's hard to define what's right or wrong. For some people, it conflicts with their moral or religious principles or beliefs. Just the fact that you have this fantasy is ok. Many people have a lot of different fantasies that they don't announce to the world, so it may make you think yours is one of the worse kinds.

As CANDC pointed out, just focusing too much on the fantasy gives it more life. And I would add not so much focusing on it in and of itself, but focusing on the fact that you shouldn't have it. What you can do is when you see this fantasy come in to your head, notice it and say to yourself that it's just a fantasy, just a thought - try observing it from aside. This will help you realize that it's just a thought, and not some monster that is going to make you do what you don't want to.

On the other hand, if you and your wife want to experiment sexually and enact the fantasy and it doesn't hurt you or the person you involve, then you may consider it as well to give release. What you described doesn't sound outrageous or something to punish yourself for.