I also have trouble being assertive and would list the exact same reasons as you. I also avoid confrontation in general which may or may not play a part in your situation. But for me, being assertive would be out-of-the-questionably difficult because of that, and if it did become a problem I would be afraid to call the same establishment again to order.

I think it is not that bad about being nice to the delivery guy. Maybe he was new and doesn't know the area yet, whatever, it sounds like he was at least trying to do his best to get there. (I'm guessing, I don't know his tone of voice or attitude while you were directing him) but it would be nice to be able to assert yourself for the fact that the pizza was cold. Especially a large establishment like Papa Johns would have policy in place for that... even if it isn't made as a marketing promise (i.e. "delivered in 20 minutes or it's free!) they would have some rules or guidelines for the managers since I would expect this to come up often.

I would guess that it would be even harder to call now "after the fact" but if this was very recent you might want to call to try to get a coupon for a free pizza or something. Maybe not a good idea as they may give an argument since you didn't call when it happened. But that may be a good goal to try if it happens again.