I am not a parent or anything, but I do prefer one grandparent over my other grandparent. That being said, I was endlessly abused by the one I distance myself from, which is clearly not the case for you at all.

Before the abuse took place, my grandmother (on my dads side) would try and buy us love. Which, with three children who grew up appreciating what they already had and time spent together was rather intimidating. I didn't like it. But she felt it was what she needed to do to make us love her more. And that wasn't true. My grandma (my moms side) never spoiled us. She loved us unconditionally and was endlessly supportive. And while I wavered between my own emotions for years, I, as a child, never mentally went "I love this one more than that one" the relationships were just different. And different is okay. Also, children aren't going to prefer just yet, their minds are still developing. And in their own little worlds, they tend to stay attached to who they're used to (like you said, the baby is attached to his mother).

I feel like a lot of tensions are formed this way. At least in my family they started like this. Children are finicky and fussy, and they need time to grow and discover and form their own unique connections with family members.