He sounds like he may be an avoidant type. The kind who tries to get close, but not too close, backs off, comes back repeatedly.

In other words it's him, not you. Especially in light of the fact that he does this with many women.

Thank your lucky stars that you dodged a bullet!