It sounds like you are possibly sort of apologizing for bringing the topic up! No need for that! You have a legitimate concern.

Truth be told, we just don't know the long-term side effects/damage of many of the psychiatric drugs. Klonopin (anti-anxiety drug) is being suspected as a culprit for some memory problems I have, so I am being weaned off of it!

As your doc said, we have to weigh the cost/benefit ratio in deciding what to do.