Not what you want to hear, I'm sure, but nevertheless...

I'm sorry you're hurting, but I'm a bit confused. It seems like he apologized several times, following up with asking you what you wanted to talk about. When you told him about the b-day party, he responded briefly. Did you follow up and say anything more about it? What I'm hearing is that you interrupted him several times, he invited you to start discussions, you didn't follow up with saying much, then decided to sit in silence, all the while expecting him to rescue you from that silence. I would wonder why he would do so when you'd shut down every other opportunity for conversation. It just sounds like you walked in looking to create a rupture (perhaps out of your anger about his doubting where you grew up?), and it would probably be helpful to think about why that might have been.