Pack his bags, call a lock smith and change the locks, dont let him back in.

If he starts shouting etc at the door call the police.

You have a baby to consider now not just you and you are right you are not responsable for him.