I really think it has more to do with the mental thing, like if you are in love and find the man exciting, and, especially, if he is a considerate lover.
I wouldn't care at all about a man's "size" if I loved him.
At the same time, looking back on my unfortunate history with sex and "love," I realize I've had the unfortunate mindset of equating sex with love...!!! which it seems was not the same for the man! Very disconcerting to realize this at my age!, now after three years of abstinance resulting in a comfortable feeling of asexuality!
I don't think I've been promiscuous but I do seem to have had a wealth of experience!
Patty