I'm really sorry that you are hurting.

With regard to the joke, I would like to note that he acknowledged he was doing badly. In other words, it sounds like he accepted your reaction and didn't try to shame you over it. I myself feel the same way: your reaction was your reaction and that reaction was, and is, valid.