As a guy I can't post from a female perspective (well duh!!! ) but I have had problems in the past with people touching me. I used to hate it and still sort of do. What doesn't bother me is being touched by those who care about me. I have no idea how close a relationship you two share but if you are afraid or nervous when he touches you then maybe you should try working on trust issues. Even though, like he said, he won't rape you, in your mind you could be convinced that it is a possibility. Give it some time and talk to him about it and then maybe you'll begin to feel really secure when you're with him.
Hope that helped. Cheers,
Peter