Has anyone ever mentioned tactile allodynia to you. I can't stand light brushing against my skin feels like sand paper against my skin. But there are variations that may be worth looking into. If you have always had issues with touch, then perhaps it's something physical/neurological.
Really I just wanted to tell you there are conditions similar to what describe. Might be worth looking around.