Personally if anyone is to close too me I will indeed back away.

But in your case if you dont think that could be the cause then maybe its something else.

Could it be the conversation? or the location? not to offend .. but personal hygiene? Especially dental/ bad breath... That is something that will indeed make back way up for.

What type of people are these ... Coworkers? causal hanging out? what type of situation?