There Is nothing wrong with this. My father and your father probably weren't exactly alike, but he was scary when I was a child.
When I got older I would get sick and dry heave when seeing a man on tv acting mean/crazy.
All of my teen years and most 20's I was a virgin. I didn't trust boys/men. I would shake and run away if a boy even tried to ask me out. At 26, and a lot of jager, I lost my v card, but turned out he was a pedophile. It took a long time to even think about sex or a relationship again. but I did meet a very sweet man. nothing like my father.

I would say if you want to take the Observatory route, there is nothing wrong with that. With what you have experienced, and in general, there is nothing wrong with taking things slow. If you like someone, and they like you, Then they will respect what and where you are coming from.