(((((((((((((((Seeker)))))))))))))))
Most of us can't choose whether or not we are going to be attracted to someone. And with the history you two share, it's only natural that there would be a strong attraction between you. It sounds like you both "know" that it would be a bad idea to get involved physically, but there's that old attraction, pulling you back together.
Now while we can't chose our feelings, we can chose how we act on those feelings. He has made the choice to stay in his marriage, and you both have made the choice not to engage in an extra-marital affair (excellent choice, by the way

). But when you are together, that attraction is so strong, and it feels so good, and comfortable, and right. It can make it real hard to remember the choices that the more rational side of you made.
I don't know anything about this man, and maybe people who are saying he's just using you are right, but my guess is, he's being torn between the smart, rational choice that he's made and this strong attraction he has for you. It makes him seek you out when he's in town, and makes him sound cold and distant with his wife when he's with you...but apart from you, from the temptation, his rational brain takes over again, he remembers that choice he's made to stay faithful to his wife, and his response is to try to "undo" what he did and said with you.
I think people don't want to believe that temptation can get the best of them. They want to believe that their brain should be able to control their urges, and that they shouldn't have to give up a friendship because there is attraction, that they can control it and not let it get out of hand. But to continually put yourself in the path of temptation is to take a big risk of succumbing to it. And from what you describe, it also causes pain to have to see and be near something you want so much, and not have it.
I guess this is all a *very* long-winded way of saying that I think your decision to stop e-mailing and seeing him is the best thing you can do to take care of yourself. Not because he's a bad guy, but because your relationship is already too intimate to keep it going without causing you both a lot of pain.
Good luck.