Can you say something like, "It's been nice chatting, but I'll let you get back to work" and then make yourself unavailable?

Sometimes the answers is to hire someone else, unfortunately.