It's not only hurtful, unconscionable, slimy and sneaky to run around behind your partner's back for several years with someone else, but then it's most insensitive to continue having some kind of "friendship" with that person. This partner of yours does not deserve you, and you do not deserve this kind of treatment. Dump him. He cannot be trusted, his character is very poor and weak, and he'll never treat you right. There's far better out there. Don't think that there isn't, or that this is all or the best you can get.