It is against the law (in many states) for someone to harrass you on the telephone. I had to handle this once with a land line. I can't say what the rules are with cell phones, but there is likely some remedy available.

I called the police after I got a disturbing, threatening phone call. They came out quickly and they called the telephone company who provided them with the location from which the call had come. (The phone company won't tell that to you.) The phone company told me what to do if I got anymore calls like that. (There was some numbers for me to dial to signal the phone company that the call was of that nature.) The police and the phone company were very supportive. Luckily, I never got another call like that.

I think every land line company, and perhaps cell phone services, have a policy for helping customers deal with repeated unwanted calls. I'ld recommend talking to them. Again, you can talk to the police about this, also. I've found that police are very ready to give advice on dealing with harrassment. They'ld like to help avert something from escalating. Also, they can tell you the limits of what they can do, which is worth knowing.

You certainly don't have to talk to this creep at all, if you chose not to. That's what I originally suggested.