I think (and this is not particularly pleasant but it will mean you are not at his mercy) you need to have the specific discussion with him about it and then decide what you will do if an arrangement acceptable to all parties cannot be worked out. The hard part is also that you are still waiting for him now.