Oh, that's so hard! I would maybe take a "vacation" and go somewhere else for as long as you can (rent a room at a YMCA or hospice/women's shelter), work to not be the "same" places as X and not let anyone who knows X "well" know where you are either.
If you own your house, can you rent it out for a year or something and use that to rent something elsewhere for a bit?

Maybe you could make friends with a realtor and put your house up for "sale" and make it look like you have moved already and get a "sold" sign, etc. all while you are elsewhere.

Or, in the immediate case, I'd make friends with neighbors so they can back you up and call the police while he is there "for hours" banging on your doors and windows. The police aren't going to ignore him if the neighborhood gets riled up :-) How long have you lived there?

You may in the end though have to decide whether your safety and peace of mind is worth more than your location and leave/move elsewhere.