I think you went a little too far in your "half" of the discussion and did not let him tell his half. What can he say to your saying you feel he doesn't care that much about you? There is no way he can tell you what he does, in fact, feel (if he has the words/experience to identify/speak about his feelings; I think many men do not) because you have already told him how you have interpreted him and he can't do anything about that.

I would have left it at, "When you did/said this, I felt hurt." He is not responsible for your feelings, he cannot hurt them, only you can feel what you feel, it's your interpretation of what is happening around you that determines your feelings. Likewise, "didn't do/say" is something I would try to stay away from because it implies that other people read our minds and do/say what we want them to?

It is all right to end a conversation with someone else, to hang up the phone or turn and walk away; however, it is polite/educated if one knows/is emotionally able, to first tell the other person one is going to take that action. Were I he, I would have said, "I am unable to continue this conversation at this time; I am hanging up the phone now."

Everyone takes care of themselves as best they are able. If you still care about this man, I would call/email him and apologize for putting him on the spot and asking him to try and warn you in the future if he is going to terminate a phone conversation as, "I do not like abrupt, surprise endings to conversations."