I had to cut out all coffee and caffeinated drinks for 4 years because they gave me panic attacks. Even now, I can't drink coffee or cola after lunch time because I get anxious and have insomnia.
Caffeine affects everyone differently. And every cup of coffee (unless you get it from the same source every time) has different amounts of caffeine in it.