It sounds like he is trying/tried to exploit you. I understand the dilemma, though.
I have encountered similar situations where I wanted to stay in contact with someone, but they had other aspects of themselves which really discusted me (or something similar.)

I dunno all the details and full history, but if I were you, I'd continue to stay away from him.