Personally, I wouldn't walk away from a comment like that. I would tell him exactly what you said here. For example, say that you are finally getting on your feet and comments like that are hurtful and counterproductive. If he says it was just a joke, tell him it doesn't matter if he was joking or not, it's still hurtful and a ****ed up thing to do, so please don't make comments like that anymore. You have to set boundaries and stick up for yourself. If you don't, it's just going to continue and make you resentful.