I think the article The Well posted mentions some of the relevant key points:

Yes, little children will have memories of the youngest years, we all do, but those memories are overwritten, lost by the time we reach adulthood.

"Memories" of those ages are often reinforced anecdotes: we see photos, hear family stories, see relics of those periods, etc. and those things all combine to form what we consider to be memories but are recreations. And if we try, as someone mentioned, to continually scrutinize something, that can actually reinforce a false impression- the mind searches for order- in the absence of order, we often try to create it. A simple example is the way I mght leave the letter i out of one of the words in this sentence and yet you all would read it clearly, no loss of comprehension, your brains filling in the blanks for you.

That's not to say no one has memories of any type before XXXX arbitrary number of days old, but that vague longings, sensations, somatic experiences, feelings, desires, and even sensory cues don't necessarily come from memory. I agree with the others who've mentioned it that regardless of source, messages we get can teach us something about ourselves and hopefully help us meet our needs and fulfill our desires better.