I think *you* are the one being abused from what you've written. Abusers often try to turn it around and blame the person they are abusing, so I am not surprised he is telling you it's your fault.
I hope you are able to get out of there soon -- it must be so difficult to have to live there while you are looking for more work. Stay safe!!