Would you be willing to talk to someone about it - maybe a therapist? Perhaps there were unspoken things you were picking up on regarding touch when you grew up as a child.