Speaking from a guys perspective, we have a tendency to blurt things out without actually thinking them through. My guess he was only make a joke, that to him seemed funny, but to you was hurtful. Talk to him about it. I've done it before, and the problem was solved by having my ex-girlfriend confront me about it and talking it through.