There is a London based charity called Kids Co and their founder has done much research on brain function and the effect of emotional neglect on developing children's brains. She asserts with passion that neglect is much worse than abuse in terms of how the victim is left to cope with life afterwards as neglect stops certain areas of the brain developing that stops the victim from being able to function on a social level in society and is therefore left 'brain disabled'. Abuse is terrible but it does not stop social development of the brain.

So, I'd see your situation as worse than abuse, especially if there are children concerned.

It is not all doom and gloom though. The brain can still be developed if the victim is shown love and attention and their emotional needs are fully addressed but it is a long process.
I'd suggest you do what is right for the children, if there are children involved.