But you never really had a relationship with this "other" woman ... it sounds like you're infatuated with the idea of the other woman but you hardly know her? Or do you? Did you two have an intimate relationship?

You're right. Your wife doesn't deserve to be second best.