I am a little inarticulate about this because I don't really understand my own emotions

, but I'll do my best to explain:
I started dating this guy in late October of 2012. In December, I graduated and moved home with my parents for a few months while I was still in the limbo of looking for work. My hometown is about an hour and a half away from where I went to college, so I wasn't incredibly far, but I was feeling weird about graduating and looking for a job for the first time and figuring out what to do with my life, so he and I didn't see that much of each other from Dec to Feb at all. And then in Feb, I got a temporary job and ended up moving to another state for three months. So at the end of the job, I moved back home again for about two weeks, and I'm off to another job in another part of the state this Sunday.
Anyway, while I was home, I visited him and we went out. Whereas before, I found his attention nice and I liked our little conversations, I found it all pretty irritating this time right from the get-go. When I look at what specifically annoyed me, it's mostly small things. First, I said something about Da Vinci Code, and he went off on this LONG tangent about how much conspiracy theories annoyed him. If it had been the slightest bit lighthearted, I wouldn't have cared, but he was railing on and on and on about how stupid people who believe 911 was a hoax are, until I was practically shouting, "DUH!" at him a hundred times, and snapping that Da Vinci Code is just an entertaining novel that I liked to read when I was 14. Then, when we were talking about the new Star Trek movie, I explicitly said I hadn't seen the old shows and I didn't care how true to the series it was, and naturally he IMMEDIATELY launches on this tangent about how inaccurate they are. He did not appear aware of how bored I was by this.
The real thing that has me concerned, however, is this: after we walked in the park for a while, he took me to his house. I thought his mother would be home, so I didn't think it was a big deal, but then when we got there and we were alone, and he started touching me and kissing me, I realized his intentions were probably to have sex, or at least to touch me in hopes that it would lead to sex. This makes me more uncomfortable than I can express. I don't want to have sex with him. I don't even really enjoy being touched, and I only enjoy kissing when it does not involve the other person trying to shove his tongue into my mouth or use his teeth. I said immediately I didn't want to have sex, and confessed that I'm a virgin.
I told him the truth, which is that I'm not attracted to many men sexually, and that I feel fairly repulsed by anything more than kissing. He immediately told me about his two one-night stands. Then he asked me about my other relationships (I've only had two), wanted to know who the people were and how far we got, and whether I masturbated often. I answered all the questions, but I felt uncomfortable. I don't feel like we've been dating long enough to talk about this stuff.
All the while, he kept holding me and telling me that it was OK and that he wanted to be with me, and that he didn't have a lot of experience with dating and how he spends a lot of time alone.
Maybe it's wrong of me, but I just feel like it's not really ME he wants. I feel like he just wants a woman's body, and I happen to fit the bill and am reasonably attractive (or at least, I'm not fat and have all my teeth...). We left it that we would keep writing to each other as I go off on my next job, and that we'd try to see each other in November, but I kind of wish we'd just broken up. I feel like we don't really understand each other, and honestly, I don't feel sexually attracted to him. I feel bad about this; he's a nice guy and he hasn't wronged me in any way, but I just don't see this relationship leading anywhere.
However, I have no experience. Does anyone more experienced than me have an advice? I'd really appreciate it, because I feel kind of lost.