I can completely relate to this, I don't have a license, a permit and I've maybe practiced behind the wheel four times...each time I panicked uncontrollably and cried. My boyfriend gets kind of annoyed as well because he just can't understand why I can't do it. How do you even explain to someone that something so many others do without even thinking causes you so much distress? I don't think I will ever be able to drive, but you've done it before and I think that's a big accomplishment! I know it can be tough but I think the fact you've done it and passed the test can give you a good edge.
Is there anyway you can take it slow or maybe get a couple lessons from someone other than your partner?