I'll go out on a limb here, but it may be a phase all husbands go through.
It is a powerful fantasy for the more sexually adventurous males....but not something they'd actually carry out.
Again, one way to possibly kill it is to act interested but in someone he may particularly not be attracted to...it's a gamble but it may put an end to it.