I agree that he is using that fear to control you. Sounds like he could not handle that responsibility anyway. I would recommend a consultation with a lawyer at least find out what legal recourse you have if he does take them. Armed with that information you can make an informed decision on how to move forward and get you and your children in a healthier environment.