It may not have been the most tactful thing for him to say, but he may have a point. I stopped 4 medications several months ago and was doing OK for a few months before I had to go back to hospital. My psychiatrist told me that it's normal to feel fine for the first while after stopping meds, but after a few months it's common to come out of remission.

I am now on two meds (an anti-depressant and one for nightmares) and am stable. I hope that you are able to live without medication but please know that if you ever have to go back on meds, there is no shame in that. I wish you all the best.