What you are experiencing is not normal. For a doctor to dismiss it as being purely a mental health issue without doing some form of testing is irresponsible. It may be from a physical cause or mental or mental issue or both. Cognitive abilities may also play their part.

Stress alone can knock a person's memory for a loop. Recently I moved house and my memory was bad to the point where I was so worried about it that I mention it to my P'doc and he did a quick test to see if I was experiencing dementia. When the stress of moving settled, and I got used to my new place, my memory picked up but it is still a bit of a sieve at times even now.

It really couldn't hurt to get a second opinion from another doctor.