Whenever I wonder something like that I think about how neanderthals would choose a mate. A male would have to be able to demonstrate being healthy, being able to provide and to father strong, healthy children. Ergo, a physically strong guy would be attractive because he could wrestle a bear to be able to feed his clan.

Men and women still react that way. Guys respond to women that physically demonstrate fertility and child-rearing while women tend to respond to guys that look healthy and strong.

Thus the reason my wife doesn't seem to find me attractive. I'm small, slow and not overly physically capable.