Wow! I definitely agree with the others that responded. This child is in serious need of some help!
As someone who has babysat for about 100 children over the years (individual families, kids in my extended family/friends children, church nurseries etc) this behavior is obviously not normal and goes well beyond the typical "testing the waters," that would be developmentally appropriate or expected.
I can certainly see why this child's brother might be afraid of him at times and I too wonder what the heck is or has been going on in that home!
I don't blame you if you decide to quit. This child could hurt you even worse next time! Also, perhaps if you quit and are honest with the mom as to why, it might be the motivation SHE needs to get her hurting child the help he needs and deserves.
May I also suggest you watch the bite area for signs of infection--which you probably are already doing lol. Maybe make sure all your immunizations are up to date also!