Hi, Grimace! In case we haven't met, welcome to Psych Central!
Ah, yes, I want to put my Freudian cap on.

I will tell you right off the bat that I am not a therapist or any other kind of mental-health professional. However, I agree with you that these fantasies come from your relationships with other women, who cheated on you. Now you on some level assume it will happen sooner or later, so in a sense, you figure why not encourage it? Then you can have control. And one thing has led to another..... That's my theory anyway.
I have not had these experiences, especially given I am not male. But, I will tell you this, whether anybody else has or not, these fantasies are concerning you and interfering with your having a normal sex life. And that means you should see a therapist to get to the root of the issue or, depending on the therapist's bent, help you to stop the fantasies more
directly. Make sure to see a male therapist.
I am saying all this because I care. I don't think you're some pervert or whatever. Okay?