I think for me it has to do a lot with boundary issues. There were a lot of people in my life who waved off 'no' and thought my retreating was 'cute'. So now all touch is suspicious and not to be trusted because I know my wants will be dismissed for someone else's. There's an illusion of safety if I'm not being touched.