Maybe its got something to do with the fact that your father was absent. You do not have the father daughter relationship as a familiar inner concept and translate it to what you are familiar with - a relationship with a man. Sex is about intimacy. Maybe you crave to have intimacy with a father and that desire / need as expressed as a sexual one. I wonder also what was your first encounter with a man like?