Sounds like this guy is just an addiction for you. I have been addicted to women before, and they are usually the worst possible people for me. They seem to fill some need that is actually just an issue from long ago. That's why it's best to just start out as someone you can be friends with and let it grow into something more passionate and deeper. The addictions will only turn out to hurt badly in the end.