Have you ever crashed your bike before? If so, I can understand your fear.

If you're anxious about crashing, what about wearing long sleeves and pants? That may help you with your fear because that could add some protection from scrapes and cuts, and having protection might make you less anxious (since nobody really wears elbow and knee pads...).

I know you're in CA, but you could always wear a flannel top and abrasion resistant pants, like Carhartt's, which is what my dad wears when he goes mowing out on sunny days (e.g., 90F weather). He says it's not too bad because the riding mower allows some breeze, so I'm guessing the breeze you get from a bike would be similar.

I actually have a female Carhartt jacket that's abrasion resistant and I have it for biking and other outdoor activities where I might fall. But of course I'm snow land where weather is regularly around 0-15F, ish, in the winter. Sometimes like -10F, lol. So a jacket probably wouldn't be a good idea for you unless you want a heat stroke. Though if you're fine with jackets -- e.g., for blustery days -- then you could try that. That'll probably protect against bruising.