Does your primary care physician have a long-standing personal connection with you? Obviously when you are in the throes of severe mental illness the specialist is the way to go, on the other hand, a generalist can see the "big picture" of your life in some ways the specialist can't. Perhaps they can offer a balanced view between the psychiatrist and the average person and help here.

Another thought: I agree with the idea of continuing therapy but in a less frequent interval.