I think that some anxiety is normal when you make the decision to taper off medication. I know that I've considered going off meds and I've spent a great deal of time wondering if it's the right decision or if I will relapse and end up worse than I was before.

Do you have a therapist? This would definitely be a good thing to talk to your therapist about and express your worries. Without being a professional, it's really hard to judge whether what you are experiencing is normal anxiety or if its the beginnings of paranoia type apprehension.