You have to my knowledge told us twice he is abusive/aggresive/not nice to you and your children.

Now they have been taken from you.

I am not condemning you as its your choice in life what you do however this is not unexpected.

Personally I would do the right thing and fast or you could loose your children for good.