Well I do know how you feel I was in an abusive relationship for years, and then left.
I am not judging you, but working on what a healthy relationship should look like, and helping your children deal with the trauma should be what your priority.
Just because someone doesn't hit you does not make them your ideal mate.
If you are afraid to open your mouth, and so stressed out he will leave if you say the wrong thing, this is way off what a healthy relationship should look like. You should be able to be free to be yourself, and comfortable and safe. You are not.
It takes a long time to get over the trauma you and your children have suffered.
Are you sure you have given yourself that time? And found out what kind of man you want to be with?
You should just be able to sit down with him, explain what you are going through, and work on a solution together, that is what a loving partner would do with you. If you can't do that, and communicate your needs, something needs to change.