That's a really tough situation, and sadly there are no precise answers. But follow your heart, and your gut feeling. If he hasn't already started any court cases or any battles, i would suggest that you could beat him to it, and it will give you more of an edge on it. because the focus then will be, keep the kids safe from what has happened and actions that have already taken place.. if he is first, it might be a little more work to get whats best for them, and you.
If anything starts to look like it might be "playing in his favor", demand that if they are to be allowed to spend time together, that it would be supervised visitation (a state worker usually is who sits in, and it's done in the place that the worker is at).
It also depends on what state it is in, and where the intial county that the case was opened in.
Once the case has been started, usually they wont allow for either of the parents to move to a different county. (my experience as a kid in texas). but with safety being an issue, they are also able to make accommodations.
Make sure everything that is done, and happens is documented by a professional. even if it's a small scratch. Please dont worry what others might think of you, you kids are more important than what a stranger thinks of you . You're very strong! And a survivor! and a great parent!

EDIT: PS if you have a voice recorder, try to keep it on you all the time, and any interactions with him, have it recording. and if it's too low to hear when its played back- dont worry, because the officers can use software to enhance the volume and get it at hearable level. And it's better to record too much, than not enough. If you decide to do this, you'll want to make sure that the recorder doesn't make any kind of sounds (beeps, or alerts or anything) prior to recording; if it does, then if there's a option to silence the sounds it makes then that would be best. if it makes any sounds, it will likely alert him to it. keeping it in a pocket(hidden from him while recording), or somewhere on you at all times would be ideal, that way you have it handy for if anything arises. There are some pretty small ones at walmart, and the counselor could possibly help out with that, if that's something you would be able to do. (some therapist have them, and some don't. but it's always worth a try . and if not, since she works a lot with court cases, she's likely to know how to obtain one so you would have it). *big hugs*