Thank you sooo much for your awesome advice!!!!

I live in rural Australia, and the town's police station is only manned once or twice a week as the 2 police officers are responsible for several country towns and share the time between them, and the nearest 24hr police station is a 40 minute drive away.
Mace and other self defence things are not available in Australia, and like SeptemberMorn said, I couldn't live with myself if I seriously hurt him. If I was to harm in anyway, even if he was on my property and I was being attacked, I would be the one answering to the law, not him. We don't have solid trespass laws here like in the USA.
I've been considering talking to his wife about what has been happening, but I'm worried that she'll confront him and he'll just get angry.
I'm starting to think that moving is my only option.

Thanks again everyone for your advice and kind words.