I came from an abusive past with parents and peers. It does make me feel both guilt and shame. It very much impedes my social and familial relationships.
I held my niece probably three times her entire babyhood. I didn't want to those times. I rarely even touched her. I was terrified of doing something wrong. I couldn't bond with her even though I love her.
All I can say is that we did the best we could in those given circumstances.
Also it's never too late to work on gaining these skills. It's never too late to tell your kids you love them and try to move forward and interact as a healthier you.