jbug, I'm sorry life is so hard for you now.
I think you should talk with your regular doctor about your memory troubles. He/she will have a bigger picture than your pdoc. There are lots of causes of memory loss--declining hormones for women, alzheimers, growing older, etc. Plus, if you are taking any medications, these could influence your memory. I was very depressed for some time, and my memory during this period was terrible! This went on for a few years and I attributed it to getting older. Well, when I finally overcame my depression, my memory improved so much! So if you are at all depressed, I urge you to work on that and maybe your memory will improve too. Also, getting enough sleep each night is very important for memory. The brain does not function well without enough sleep.
I think the question about whether to give up or not is a serious one. Can you discuss with your therapist before making a decision? Hang in there and TC.