If you don’t want to confront him directly you could play “dumb”. “What do you mean? Why would you say that? What are you referring to?” Ask these questions every time he says that. Eventually he’ll stop saying it.