I have anxiety and do drink caffeine. I can go a couple days, without. Started drinking it, after my second son,was born. The fatigue I experienced was unbelievable, never mind recovering from too much spinal injection for that delivery and my MS. Caffeine, a cup or two, helped.

Your reaction to coming off, does seem extreme. Some would say, a little, in moderation is appropriate.

Hope you get some medical answers.