The worrying and panic might work in there because at that age (4) you're vulnerable to "who will take care of me" and the arguing would make you anxious someone was going to leave and the family fall apart. Been there, done that :-) You are feeling it now because it is "safe" to do so, you're not a 4 year old anymore unable to influence your own situation.