Please just give him a firm no. Don’t make excuses. So what if you end up coming avross as ‘rude’’. He’s rude for not leaving you alone.
Tbh, you might be giving him mixed messages with your polite behaviour with giving him a chance at one date even though you weren’t interested in him.
Your feelings matter too. A giy like that could possibly stalk you or date rape you.
Say no for your own safety. Block his number. Lie & say that you have a boyfriend or thst you’re engaged.