Tell him you will take it to your lawyer. Then do nothing. You do not have to jump every time he says boo.

You know you can prove that he DID know.