Sounds like this comes from his insecurities. As he sees he can trust you, this should work itself out.