First off if you go after him you'll be the one in trouble not him.

Secondly if you break anything you're looking at assault and battery and the punishment depends on your age. Regardless that is a very stupid thing to do. You need to keep your head on straight here and handle this calmly.

Walking away was good. But if he does it again you need to firmly tell him to stop. FIRMLY. EYE CONTACT. CLEAR AND STRONG VOICE. STAND STRAIGHT. Body language is just as important as verbal. I'm not gonna lie - since he's gotten away with it before he might try his luck so be ready.

If he touches you after you tell him to stop then it's time to defend yourself. Push his arm away from you if he tries again. Remember that violence is always a LAST RESORT. You DO NOT start fights but you do defend yourself.