Yes I think you're right in leaving. Often emotional/mental abuse can escalate to physical abuse. I understand you're attached to his daughters. If I were you I would be diplomatic and speak to the girls mother and see if she would be willing to let you visit occasionally. I bet she could tell you a lot about why their relationship failed. Don't go back to him.