Discretion is the better part of valour.

You did the right smart thing. Obviously that guy was looking for a fight. You probably frustrated and hurt him more by not giving him what he wanted.