Do you want to share what made you uncomfortable? Regardless, I think telling him about your trust issues is a good idea. From what you wrote, he doesn't seem very supportive.