When I am unhappy in my marriage I think of my past loves and probably idealize them. I think you should treat them as separate issues.