well first of all there is nothing wrong with you!
this man is exactly what you said-a committment phobe. there is no percentage in your remaining involved with him in a romance way. no matter how comfortable you felt with him he's not going to change his ways and overnight be the companion you desire. no way, no how.
i don't know why he's this way. clearly he has major issues.
i have an older brother this way. he's 64 now, broken many a heart, and remains single. his life could've been enriched by so many lovely women he's dated. i don't think this guy can change either. your description of him sounds like he's even guarded in social interaction. you deserve better than this. i'd move on and hopefully you'll find "mr. right".