I think you handled yourself really well. You have been appropriately assertive with that guy. What happened when you were outside with him and the other guys stayed inside? He did those scary things and... then what happened? Did you go back in the room? Did he leave? How did that part end?
Can you talk to the other guys that were there and ask them to not leave you alone with him again? Explicitly ask them to help?