All I can say is I work with databases and I see how "professionals" handle confidential personal data, and there is SO much risk there. I just don't trust "the system" enough to secure a monolithic data store with my medical info, in part for the reasons you mention.
The current system has its own risks...for example, small seat-of-the-pants companies get into the business of managing info for doctors' offices and do some really dumb things with the data because they don't know what they're doing.
But honestly, I feel better with the current fragmented and slipshod system than knowing that my entire dossier is in one fat database somewhere. All it takes is one disgruntled worker, one screwup in the security procedures, one botched employee background check that lets a bad apple in...
Sorry, off the soapbox now.