My memories of childhood are like swiss cheese... big holes here there and everywhere, good bad and indifferent. I don't know of anyone who has a seemless timeline of their life. I think some times our minds need to sort what to store and what not to store, what to retrieve and when to retrieve it.
Memories, like my son, like to play tag... the harder you chase them the faster they run. Sit patiently and wait... If they are to come back they will come in the calm not in the midst of a cross country chase.