well, i think change can be hard on a family even if you left for good reasons.... and this current living situation will not be permanent if you dont want it to be. just stay focused on becoming financially able to get out....
i am curious how the visitation schedule was arranged... was it voluntary or by court? do you have to see him on those days to pick up/drop off the kids?
i hope you have an attorney to help you through this...
in my state, divorced and separated parents are required to take a class about certain rules such as no talking about the abuse, no putting the kids in the middle of an argument, no talking or asking questions about the other parent, etc....
there are also certain limits on visitation if there is a history of documented spousal or child abuse.

and there IS a light at the end of the tunnel... you are in control of the switch to the light, more than you realize.
i dont mean any of this is your fault, i just mean that you are able to improve it.