I would say to him: Why do you persist in doing something that you know upsets me? Maybe he wants to aggravate you for some reason