I realize this is a frustrating situation, but you did the right thing by walking away. If he looked like a body builder, he could have over powered you and probably wouldn't stop with one punch. They're all idiots and were looking for a fight and a victim. The fact he was yelling at someone else shows it wasn't you/friend - don't let it bother.