I think you should go to the police, tell them what he said, and let them know you're worried that he or his friends will come after you. They might have better advice than just a restraining order, although you should get that, too. The restraining order, if for no other purpose, will show you took action against him, if you're ever in court in the future, and need to prove he was causing problems for you.