I'm confused, as I always am when I'm learning about these lifestyle choices, because if you took your vows - and discussed your vows- and agreed on your vows with your husband, and those vows included protecting and building on the unity between you and your husband, why wouldn't falling in love with someone outside the marriage be breaking those vows?
Why be married if you are still sticking your toes in other waters?
The lines are really blurred and it sounds like you were having a great time as long as it was going your way. You're completely railroading your relationship off the track because of sex.
You seem really selfish about your ideas of relationship, and like you aren't respecting the way you two had agreed on the way it should be.
If that's the case, as you described, you are just a cheater and should quit the idea that you're being faithful, what you described is not being faithful.