you could always tell him how you are feeling (you could omit the part about the future relationship) and just discuss the working together outside the doctor-patient relationship. then you could ask him if it causes a conflict, and should you maybe seek another doctor in order to keep working with him.

dang, that sounds so simple but i think would be very hard to actually do