You are being wise by not engaging him IMO. He should not believe you are afraid of him because often that gives an abusive individual a sense of power where they may pursue further. By not engaging "you" have the power and that is what you need to maintain.