I would probably tell him, and then he might understand, and reassure you. Help you try and get over the issue, so that you guys can have a healthy relationship.

Good luck