Personally, I wouldn't let them come over either. You can never be too careful with kids. What if they took it as joking and then he really did do something to harm them? Then everyone would be asking "why didn't anyone take these kids seriously?"

The fact that he torments his own daughter kind of sickens me. Abuse can be in more ways than one. If the kids don't want to be around him, maybe that should be a sign.