I too have trouble with memories from childhood, but some of it is from the abuse and the fact we moved so much. Also, I have had ECT and it has had a negative impact on my memory.
It sounds like a lot of people don't have strong memories of childhood, so you're in good company.