That sounds really stressful. Have you ever explained to him how difficult this is for you and asked for better communication outside appointments? If the situation isn't going to change I would suggest you definitely find another doctor. Having contact between appointments is crucial for proper treatment. You can't be left hanging like that.