It really depends on what meds your taking.. Do you have enough self awareness to catch your unmedicated episodes coming on? Do you have a trunk full of coping skills and know when and how to use them often?
Do you have support of family or /and friends that can let you know if something is "off"
Will you still be able to have that Pdoc to help you if you wind up a hot mess?
There's a lot of things to think of before you toss the meds. You have a pretty strong history of delusions and aggression that's worrisome. Did those things happen while you have been on medications? I'm not a fan of Big Pharma, But you have to look at this with common sense and logic.
Take care

and Welcome to PC