I too, do not like to be touched. My husband is very understanding about this. I have to make it a point and remember to hug him and hold his hands. When we do touch I feel better if it is on my terms. If he doesn't let go, I begin to panic. It took us a long time to get to this point and I am thankful that he understands. This is a result of my childhood, nonaffectionate parenting or when there was contact it was abusive or just cruel i.e. torture tickling.
When we sit together or if he walks past and brushes my back and it makes me feel uncomfortable, I gently let him know and he understands.
But I do have to remember that his family is very affectionate and he needs physical affectionate. I consciously make myself hold and physically love him. I am getting to where I enjoy and want it more often now.