I think you did the right thing. If the guy got lost then he got lost. He called and got the directions, what more could he have done? Delivery people don't know every nook and cranny of a neighborhood. He may be new and still learning. You said yourself you don't give good directions and maybe he got confused. It happens.
If it happens again, just take a dollar off his tip and smile pleasantly as you say thank you. I agree with you with the not pissing off food servers. I don't think it's a good idea. You know what they say about getting more flies with honey then with vinegar.
You did the right thing. Stop second guessing yourself.