memory is always a strange thing and different for everyone. i grew up with three older sisters. one of them doesn't remember things before the age of 9 (before i was born). i don't know how the oldest one's memories were though regarding childhood.

for myself, i remember school and friends and the insides of houses i lived but not much about how the household was in terms of interactions with family members until i was 11 and my household changed for the better.

in saying that though, there were also a lot of things i didn't remember, specifically trauma, that my mother remembered or things i remembered that others didn't or i got one small part of it wrong but the rest was right.

memory gaps for me have always been confusing, and i've always had short and long term memory issues including while i was in school and even just what day it is or what month and think something happened a year or so ago when it was longer.

like others suggested, it might be helpful for you to talk to a therapist things are distressing to you.