I would either try harder with my driving so I could eventually get to his house or I would stop talking about my driving problems with this friend if the critical aspect makes you feel bad. Sometimes critical can act as a goad or challenge and help us but it sounds like it is distressing you so I wouldn't talk about it with him.

I assume, since you and your grandmother took taxis everywhere that someone else driving is okay? I would cultivate a real life friend who could drive a short route, then you drive it with them in the car, then longer, etc. until you are more comfortable. Perhaps seeing/feeling it being done and having someone with you while you attempt would help? Maybe you could take driving lessons at a school? Even though you know how to drive, having one of their cars that they can control if you panic might help, kind of like training wheels? I'd look into something like that if you don't have a local friend. But I don't know if you want to get over your driving anxiety enough to put up with the anxiety while you're doing that? If this friend isn't that important to you, whether you can drive to his place or not, then I wouldn't bother? I'd go on with whatever you are doing to get over your driving and social anxieties. You do not have to defend yourself to anyone, it's your life to run as you see fit, not theirs.