Folder Scattering
Metadata
The Folder Scattering component ensures that files are evenly distributed across the file system to prevent over-utilization of any specific storage area and allows for the extension of storage capacity without interruption to ongoing operations.
Let us assume the following AIPs:
../storage/aip
└─ aac2c410-ae70-446a-a5ff-0b8396119201
└─ aac2c410-ae70-446a-a5ff-0b8396119202
└─ abc2c410-ae70-446a-a5ff-0b8396119203
By using folder scattering, we can achieve the following division of storage (hexadecimal notation).
../storage/aip
└─ aa
└─ c2c
└─ aac2c410-ae70-446a-a5ff-0b8396119201
└─ aac2c410-ae70-446a-a5ff-0b8396119202
└─ ab
└─ c2c
└─ abc2c410-ae70-446a-a5ff-0b8396119203
- First layer is composed by the first two digits (256 possibilities)
- Second layer is composed by the next three digits (4096 possibilities)
- Lastly, the folder named after the full uuid.
As the AIP identifiers are UUIDs, which are randomly generated, the number of AIPs per folder will equally distributed. This process ensures no directory will be overwhelmed with a large amount of sub-directories or files, which can become a problem when the quantity of AIPs is above the tens of millions.
By dividing the storage into equally distributed pieces, this process also allow storage to be spread through several heterogeneous storage systems. This can be needed for coping with backup and restore processes on very large and time-critical systems, where storage is tens or hundreds of Terabytes of size and the acceptable time disaster recovery process is short, so restore needs to be done in parallel or focused in a specific piece of the storage.