Maybe he doesn't know what to say or do although he knows your hurt. It was rude of him to talk on the phone when you were there spending time with him, but he did apologize and seemed sincere. I wouldn't dump him over a mistake like this.