I don't think anyone was suggesting the possibility of infants being terrible...

I think we're discussing behavioral cues that are easy to interpret as offputting, such as turning, pulling or kicking away from a caregiver, extraordinary amounts of screaming, scowling, non-responsiveness, etc.

There's just a spectrum of infant behavior and each child occupies a little different place on it, with those at one end unfortunately having a correlation with worse treatment. That's what I was taught studying child psychology. I didn't find it pleasant, but I came to understand that it's not the same as saying those infants are in any way bad or even remotely responsible for any mistreatment. Correlation is NOT cause.