I would suggest being very honest with him, at this point. Explaining like you did here. Maybe even sharing like you did here?
There's a chance he'll turn from you, but there is a chance he might accept this.