In the following scenario (for demonstration purposes), a disk must contain 500,000 tiny files (each exactly 2 kilobytes), and 500,000 large files (each 32 megabytes), totaling altogether 16.001 terabytes:
Let’s pretend the chosen file-system is exFAT (which supports an Allocation Unit Size between roughly 2 kilobytes – 32.768 megabytes).
If the disk is formatted with an Allocation Unit Size of 2 kilobytes (smallest exFAT option), it unfortunately reduces I/O performance of larger files.
If the disk is formatted with an AUS of 32 megabytes (largest exFAT option), this ends up wasting tons of space since each tiny file wastes 31.999 megabytes per file; all million files waste about ~32 terabytes of wasted space.
Why hasn’t any manufacturer designed a disk (or some firmware) that partitions a disk into 2 partitions, each with a different AUS (one with the smallest, one with the biggest)? Then, this disk can automatically manage (without OS intervention) file changes, keeping larger files in one partition, and smaller files in the other? Both partitions could look like a single drive, or spanned drive? Or is this basically the concept of compressing files into a .zip?
This could be simulated with a massive .iso disk image, or VHD residing on a disk formatted at the highest AUS level. The large files can reside in the disk, while small files can reside in the virtual disk image. Many OSes can mount the disk, with this disk being formatted with the smallest AUS.
What are some technical setbacks, or explanations why this wasn’t done?