There is no need to continue to speculate about the memory. What's going on right now with your father is abusive. You know what you know now and what you know now is unacceptable. At the very minimum you should discuss it with your therapist ASAP.