I was able to wean off Klonopin over time and now take xanax as needed. I do think something has affected my memory ability, as I am pretty much a basket case. But age is involved, too, now.
My suggestion is to take the minimum you need to cope and in addition, get therapy for anxiety.
I've taken some version of these meds for years, and if you can, as I say, just take as little as you can. If you need it long term, then take it!