If you had a history of panic attacks BEFORE taking lithium, then the panic attack is probably not induced by the lithium. Anxiety is not one of the common side effects from lithium although a rapid heart rate would be.

You may be going through a stressful time in your life and it is definitively easier to blame medication for physical problems instead of the trigger in your life. I would talk to your doctor before quitting any medication if you have any concerns. A lot of the medications they prescribe have nasty side effects and can actually induce remission if not done with some prior knowledge of going off the medication. Usually your doctor can give you a plan that allows you to safely get off the medication to try something else, hopefully with minimal side effects.