That is verbal abuse. I would tell them that when they say that, it is very hurtful; more than likely they will argue (abusers do that), and once you have told them, you can decide to not have a conversation or relationship with them. That is cruel.