I can see how someone could hold things together for a while, go on vacation, and then come back and not be able to deal with the crazy schedule anymore.

If he was upfront from the beginning about wanting to take things slowly, it's hard to imagine that he is doing some sort of slow fade on you. Does he give any indication that he is the kind of person who would do that rather than just tell you flat out?

Also, I am not sure if I am misreading this --- has he not contacted you at all after telling you he was too busy for the next few weeks? How far into the three weeks are you?