Ickydog, I'm sorry to hear that anxiety is having such a big impact on your life. It sounds like it might be worth it to try the zoloft to get some relief so that you can live your life focused on positive things (like school) rather than avoidance of anxiety and panic. 50mg shouldn't cause you problems with your studies, but should help take the edge off your anxiety.
As for withdrawals, some people have no trouble at all, for others it is difficult. For those that have trouble stopping, the main factors are how high a dosage and how long you've been taking it.
Good luck getting control of your anxiety! You've beaten depression, even though you needed a little assistance from meds, and you can beat anxiety, too. Let the meds help you until you don't need them. Then you can deal with weaning off. I hope you'll do fine and that you'll find it even helps with your schoolwork!