Have you had anxiety for years? I'm not saying that your anxiety is completely caused by caffeine; but it can't hurt anything to wean yourself off caffeine (when you're feeling better) because it IS known to exacerbate anxiety.

Are you in therapy?