I think I would be hurt too but the more I think about it, this kind of joke is sort of a diffuser of tension. Sounds like he was talking about patients collectively and not you in paticular. Maybe he is using the moment to be a"real person" and let you in that he is human too, and maybe having a rough day and feeling "outnumbered" by his clients. In a weird way it is a moment of intimacy.

I'd still let him know you were a little hurt, maybe he can be more clear about what he meant!