You lost your innocence when he SAed you. You were SUPPOSED to be innocent. He introduced you to a world you should have known nothing about, because you were a child.
He is responsible for what happened with your younger brother, even though you feel like it was you.
Children who are sexually abused have no manual to guide them. They do what they can to get by in uncharted waters in a world where a perpetrator is telling them wrong is right and right is wrong. Physically, intellectually, morally, sexually, and emotionally children are not yet developed enough to deal with complex issues and problems like sexuality and abuse. That is why they cannot be held legally responsible for the choices they make.

"Primary functions of the prefrontal cortex involve planning a person's response to complex and difficult problems. The prefrontal cortex resides at the front of the brain, and it is subdivided by the brain's right and left hemispheres. It acts as an "executive" for the decision making process, weaving past events to present experiences in order to make the best choices. The cortex develops slowly, finally reaching maturity in a person's early to mid-20s."

The true character of a person is revealed once their prefrontal cortex is fully developed. Your guilt and self blame reveals that you never would have abused your brother if that perpetrator hadn't first abused you. I bet you have never SAed a child in adulthood either. What happened was most likely not a reflection of who you ARE but of the circumstances you found yourself in.