Index jobs for AIND metadata in AWS DocumentDB and S3.
AIND metadata for data assets is stored in various places and must be kept in sync:
- S3 buckets store raw metadata files, including the
metadata.nd.json. - A document database (DocDB) contains unstructured json
documents describing the
metadata.nd.jsonfor a data asset. - Code Ocean: data assets are mounted as CodeOcean data asssets. Processed results are also stored in an internal Code Ocean bucket.
We have automated jobs to keep changes in DocDB and S3 in sync. This repository contains the code for these index jobs.
More information including a user guide and contributor guidelines can be found at readthedocs.