How about: I enjoy chatting with you, but when I spend too much time visiting I find I don't get everything else done that I need to in a day, and then I feel a little overwhelmed (stressed, tired from trying to catch up, whatever). Why don't we schedule a time...

Or something along those lines? My thinking is a little jumbled right now because I just woke up from a nap (sorry, was that cruel?)