I was thinking a lot about this. The more I think the more I am convinced that this guy is at least rude, unless you were calling him at 3 am when he said you were annoying. Or he is rude or something is going on with him. When younger I met more than one hysterical men that liked to lure others ( into friendship or more than that) to refuse them later. There are women like that, too. Pretty immature.
About the fact that rejection hurts, it is not only you who are in pain because of rejection. It is very painful for every one. You have said you are timid and that sometimes social situations overwhelm you, but anyway perhaps group therapy would be beneficial to be able to confront your social issues in a controlled environment. I am not sure but it is just an idea I am having.