Drop folders
Metadata
The drop folders component facilitates the automated, unsupervised ingestion of submission information packages via shared folders, which is crucial for a smooth integration with other data production systems.
When automating the transfer of records from a data production system, it is crucial that information can flow from the data production system to the archive in a scalable and decoupled way, while ensuring information integrity, resilience and fault-tolerance. Information to be transferred can be prepared using strong formats (like E-ARK SIP or BagIt), that are capable of transferring large amounts of complex information and ensuring completeness and integrity by the automatic creation of manifests with checksums of all files.
The first step is export information from the data production system in a standard format or you can use our custom development service to add support for ingesting your custom submission format. Off-the-shelf, RODA supports E-ARK SIP and BagIt standard formats. You can easily create these packages using the command-line tool and Java library Commons-IP. Alternatively, you can use the RODA-in desktop tool to arrange your files, create descriptive metadata and produce the SIPs.
Each package should have a single record, and can have multiple descriptive metadata, multiple representations (e.g., Microsoft Word and PDF), representations with complex data (multiple files and folders), additional documentation, etc. The tool will automatically create manifest files and wrap the package in a compressed file.
The second step is to create a folder in your shared drop folder to organise the ingest batch. The name of the folder will be the name of the ingest batch submitted to RODA. After copying all SIP files, you create a special ".ready" empty file that marks this batch as ready to be submitted. And that's it!
Multiple drop folders can be configured per RODA instance, each with different users, permissions and quotas. A drop folder can be configured with specific ingest workflow and parameters, defining the SIP formats to be supported, the collections to be selected by default, the ingest steps to be taken, and the ingest notifications to be delivered at the end of the ingest process.
A ingest notification can be delivered every time or just when failures occur, and it can be delivered via several protocols:
- Webhook: call to configurable REST-API, usually from the data production system that produced the SIP and submitted the ingest batch, with a payload message that fully reports the success and details of the ingest procedure. This method is usually employed when the data production systems does an post-archive task, like marking the record as archived, pruning data or even deleting the record.
- Email: an email sent to a configurable set of recipients with a summary of the ingest report and links for more details. Usually, this method is used with the option of only sending when ingest failures occur.
- File: using the same shared folder used to send the ingest batch, a similar organisation is used to provide the ingest reports of the submitted SIPs, which can be monitored by the submitting agent.
You can follow the ingest and see detailed reports anytime in the RODA Ingest process page. This page automatically updates and provides a view of all ingest processes, if there are any failures, and detailed reports of the causes for any failure. With a fully integrated ingest transfer, your information will flow from the data production systems to your archive smoothly and seamlessly.