Our early life teaches us to be avoidant. I don't think we are born that way. The bullies push us back repeatedly so they can feel superior. This reinforces our need to retreat. I see it a bit like they steal some of our confidence and leave us feeling less.
Social skills are learnt and we have not had the chance to learn them. Could you practice some assertiveness? I know this is on my list to improve at.
Sending hugs