There are many things that can cause memory problems. Everything from emotional state to MI to drugs. If psych drugs cause brain shrinkage then a lot of us here have very tiny brains from being on them for decades.

Do tell your doc about your issues. He can tell you more about your condition than we can.