Actually, I've read that fantasizing about your wife with someone else raises testosterone levels in men and increases sex drive. Some kind of primal response to wanting to be an alpha male and be the one to do the breeding. So as long as it doesn't cause malice or feelings of jealousy or such, it's not only natural but ... again, fantasized in a healthy way ... but can be beneficial.

(I think I read that in GQ or Men's Health or something, wish I had a link ... but ... sorry.)