Oh I would stick it out. The connection you have built with him, the trust issues and all that are well worth the little wait, don't you think?

Doctor's are people too. They have dysfunctional families and real life issues with plumbers and such that take time from their office lives.

Since you've been seeing him so long now, why not ask him if there's a reason for his change in response lately. Maybe he will want you to call back after a few hours to "remind" him? Mine always wants me to call him again if he hasn't responded within the hour, as the page might not have gone through.