I would say it's important to think about how remembering would help or hinder your healing process. The question isn't "Do I want to remember?" the question should be "How would remembering serve me?"

It sounds like you still have a lot of guilt and shame about what happened to you and I would agree with a therapist who wanted to work on that, rather than on remembering what occurred. You don't need anyone to validate the way you feel or the emotions that surround your childhood memories. You're the authority on your experience. You can trust and honor your feelings without recalling everything.

I was the victim of sexual abuse starting at age 3 (as far as I can recall) and my therapist told me that it's possible there are more things that occurred that I don't remember and will never remember.

I too have felt like I'm on the outside when I meet other survivors with vivid memories about what happened to them. My memory is more foggy, I didn't understand what was happening and I was incredibly young. But that doesn't mean we aren't all survivors, that we're wrong or somehow less credible.