I couldn't drive until I was 35yrs old because of my anxiety/panic. One day I just decided enough was enough, called a driving school, finished the course, took my exam, got my license, and started driving.
My fear wasn't completely gone but slowly day by day I gained more confidence.
Now with the addition of buspirone I have very little trouble at all...I'm still a bit shaky beforehand but once I'm behind the wheel I'm fine.
I guess what I'm saying is decide, make a plan, and just throw yourself in head first.
Give it a good try and Good Luck!