It sounds to me he is afraid that his family and his friends ( who you have not met yet ) wont like you as much as they liked his ex .
I think that is causing the fear ..

I just hope you don't get hurt by waiting for him if he is not 100% sure if you are the one , and I think when he said that to you it kind of made you rethink am I wasting my time with him ?

You might have to wait another year , don't know if he had said you are the one then it would be sooner .