I can understand your thinking. I had one "manic" episode at the age of 17 and have had nothing since, looking back on it the explanation may have been severe OCD with some psychosis.

That being said, please be careful as you go off your medication. Keep track of your moods daily. When I stopped 4 meds at once a while back, I didn't start feeling the effects until a few months later and had to be hospitalized for a short while to re-adjust to a new medication. At the moment I'm on 3 meds and although I'd rather not have to take any, I know that they help me and there is no shame in taking medication for an illness. I hope you are able to stay med-free but please don't beat yourself up if you end up having to go back on something.