I would call it off before you take that step. Many people wish they would have had the clarity to see someone was not the one for them before they got married. Marrying someone should be contingent on them making you happy. If the only time you are happy is when he is not around then I think you have your answer.