I was the same way for a while. And actually I feel I was attracted to men that reminded me of my father. It has lessened over time. I know several people like this too. I don't think you need to worry.