Oddly enough, I find people standing near me more aggravating than people touching me. If anyone does cause me to become irritated from a mere touch, it is because that person, specifically, is "dirty". Please don't misunderstand "dirty" to mean I despise the person; that is not the case. My own father and, to a lesser extent, my grandmother fall into this category of people.
Are you aware of the reasoning behind why you don't like to be touched? Is it that the sensation bothers you? Do you fear they may harm you? Is it more to do with contamination?