I take Wellbutrin even though I've been diagnosed with Generalized Anxiety Disorder. While on it I have tapered down so much on Klonopin that my pdoc put me on a minimum amount of Xanax.
I guess different people have different reactions. Could it be other events in your life that also cause anxiety? I'm not one to vary much from what my pdoc says, but if you really think it's the Wellbutrin, then I guess you could do without it for a couple of days to see.