Sibelius5, I really appreciate that you've brought up this topic. I think it's something we all struggle with on some level (that is, those of us who are willing to be very honest with ourselves).
Don't forget that as a child and young adult, you didn't know how to relate to others except through what you observed in your family. You couldn't know. Your parents programmed dysfunctional ways of dealing with low self-esteem, stress, etc., into your brain, literally. Research on brain development shows that the brain learns from the environment, and in abusive environments, it learns destructive things.
I don't think we should blame ourselves for hurtful behavior as children or even as young adults because it was before we knew what was going on and could change it. Changing the program, so to speak, within the brain takes a lot of time and effort and can only be done with full awareness of the process, so you were "innocent" in the sense of not being able to do that in your youth.
It's difficult to know how to handle relationships with siblings under these circumstances. I think a lot has to do with the sort of relationship you have with your brother. I went no-contact with my sister for 10 years, and when we reconnected, it was an emotional situation, so as part of it, I apologized for letting her down in the past. It fit the situation. If you apologize and it comes out of the blue for him then that could actually make him feel uncomfortable (clearly not a desirable outcome).