The way I solve that "can I come with you?" problem is to a) make it clear that I don't slow down and b) follow through on not slowing down if they persist on coming along. It doesn't happen a second time because I am a fast walker/climber and I can't bear an ambling pace. Fast walking and hill climbing also cuts down on talking. If they are a super chatty person, I just say I need the walk to do work in my head, and they back off. I do creative work and walking helps me organize my thoughts.