Fear. Fear that they might be "crazy". My extended family doesn't the same thing. They generally refer to me as "mentally unwell" or as "sick." And, yes - every single one of them has major issues and/or a mental illness that they are in denial about.
The end result has been that I have minimal contact with them. I don't like being disrespected by people who direct their fears at me.