I think it depends on the nature of your marriage. I married someone I didn't love, just a friend, and stayed in it unhappily for over 20 years. I did the same thing...thinking about someone else during that whole time.
Patty