If he is there, simply leave; let your parents know ahead of time that if they do this, you won't be there.
No, don't get physical; he could turn around and have your arrested.