Should I worry?

First off, I should probably start off by saying that I have had several issues in my past, when I was 6, I was in a severe car/train accident that killed my little brother and a family friend, and nearly killed me, and I spent the next nearly 20 years subconsciously punishing myself b/c I survived while they didnt. Then, in about the 5th grade, I started having complex-partial seizures, which I've had, up until roughly 7 months ago, but I digress.
The thing thats been eating away at me is, my whole life, especially my life growing up, I have felt disconnected from everyone else around me, my friends, family, I've never even had a "meaningfull" relationship with a girl. I have always been interested in sex, like fetishes and such, but I have never had sex. That is probably the thing that bothers me the most actually. Plus, I have tried even, this woman picked me up in vegas one night, and we went back to my room, but when I got back up to my room, I dont know why, but I couldnt. Do I need to talk to my therapist, or my doctor about this? Is it possible that something could have gotten damaged in the accident, and its still affecting me now? Please help.

It sounds like you have tried to deal with this issue on your own unsuccessfully. It is time for outside professional intervention. Start with your T Or doc and go from there. Much luck to you.

Sounds like you are afraid to connect with people due to the inner fear you carry around that IF you care/love again it will only hurt when you loose them... counseling is in order here.

So the whole reason that I "couldnt", my first and so far, only time, all stems from fear of losing someone, if I start to care about them?

Yes... it is possible this is the main reason.
Have you been to see a therapist to help you work through your wounds?

P.S.
I remember a time after my daughter had passed away and I had a mental breakdown that lead to me attempting suicide, not once but two times, and my husband still coping with the loss of our daughter did the only thing he could to survive at the moment - he pulled away from me emotionally out of the fear of loosing me... mentally he could not handle hurting all over again.

I am currently seeing a therapist to help me work through some things, although, I have never brought this up. This is probably one of the most touchy issues I have, IMHO.

The therapist is a great idea!

I found writing it down on paper and giving it to my T to read during our session helped to open the door for a good discussion once she knew of the subject at hand.