If it gets bad, call the darn cops.
I have been bullied for years because of delusions...so I get what you say about being scared.
Grab some kind of weapon (For self-defense ONLY, a video recording device and a phone and position yourself in a tactically advantageous spot in case the worst happens.)
Or go full Home Alone on the jerk...