Maybe get a free consultation with a lawyer in regards to all this in case he does. Explain what you've explained here, including the part about him threatening a lawyer, and see what they say. They would have a better idea in term of courts and stuff should he decide to get a lawyer. Maybe if finances are problem in the event you end up needing a lawyer, try finding a legal aid place and talk to someone there.