It does not sound insane to me. It sounds like these flashbacks (whether true or not) have made you subconsciously wary of him, distanced you from him.

Do you have/could you have a therapist with whom you could discuss the flashbacks, and your relationship with your dad?