Nams, I can hear that you feel really stuck.
Yes, saying 'no' can be really hard. I have often felt this in quite a few situations and then also was mad at myself afterwards for not having stood up for myself or made my voice heard or made my wishes clear.

I agree with what the others have said here. It's crucial to set boundaries, especially in this case. Not only has your daughter been threatened, he is obviously putting an enormous amount of strain on you and the rest of the family.
You also very much put your own health at risk by not saying 'no' and having him over so frequently and for such long periods of time. It would be hard for anyone to cope with this, but it's even harder for us all on PC, as we do already struggle to keep our mental health at an acceptable or manageable level.
It concerns me that you put yourself through this, Nams.

If you find it easier, yes, as Rohag suggested, you could possibly try and write your thoughts down for the boy's mother and get your message across in this way.
Alternatively, maybe you can just briefly say you can't have him over at the moment without having to go into long explanations if you see her face to face? I know it's difficult.

Please, try and put your own health and wellbeing and that of your family first here.