It's exactly as I thought then. It sounds to me like he probably has a similar problem with making (and keeping) friends. If that's the case, he might be less likely to leave you because he knows what that's like.
I can't promise anything, but I really do think you should give him a chance.