If I remember correctly, you don't see your therapist often. I think it might help to see your therapist more frequently.
It is always up to you whether to take meds or not. The reasons you give are the reasons I stopped taking meds. I stayed off for a long time, then recently tried again to see what the effects would be after a few years of therapy. The effects were the same as you describe. For me, I decided to discontinue, and take this as meaning I have more work to do in therapy.
It is understandable to want to feel better, and as soon as possible.