Trust your feelings OP. If you don't feel safe and comfortable with him then you are well within your rights to leave. You don't have to have a cogent analysis of why you're leaving in order to leave. You can just leave because you feel like it. If you change your mind later and want to go back, you can go back. You don't owe him anything. Remember, he works for you. You are paying him.