Often when looking back to a bully experience typically when younger the victim honestly didn't know what to do about it or how to stand up to the bully. There was no rescuer or a source of comfort and that typically makes one feel unsafe and vulerable.

It's not so much the person, but the way that person made you feel that can get triggered. Ironically, bullies tend to be weak and vulnerable inside and they engage in bullying so they can conquer that inner weakness or need to feel they exist and have relevance. Some bullies are actually bullied or even abused by a parent.