Welcome.

By not doing anything keeps the ball in his court, and gives him the control he demands.

IMHO you must do a few things: notify the police. You need to have it documented with them, and a case number for you. This way in case it does become face to face and a problem, they won't just give him a warning, but will already have grounds to remove him from your face! It gives them tools they need to help protect you.

you must notify him in writing that you take his contacts as harassment, stalking, etc and ask that he cease and desist immediately. If he doesn't then you will be forced to take it to a higher level, for your own protection. Make sure you keep copies of the notice, send one to your lawyer. (Or you might want to have your lawyer send the letter on your behalf.. but that'll be at least $100.)

you need to learn to protect yourself. Take some martial arts course, learn to carry a firearm and do so, or at least carry a short bat of alluminum you can buy in the sporting goods/fishing store. A license for a concealed weapon would be nice too. Make sure your home is safe: lighting, alarm, carry your cell phone ON you...

Keep documentation of days, times, events, items etc. It has a lot of weight if you do end up having to take him to court!