I think you gotta do what feels best for you. To do otherwise would not be in your best interests. If that means telling him how you really feel, so be it. If you enjoy the friendship, then keep it as such.