Skip to content

Experimentation toward producing zarr manifests

Notifications You must be signed in to change notification settings

dandi/zarr-manifests

Repository files navigation

Just a place for experimentation -- don't judge!

The idea for manifests layout is to be consistent with our /blobs/ on S3 where we provide a few levels of "declutter" subfolders from the first letters of the zarr UUID, e.g.

128/4a1/1284a14f-fe4f-4dc3-b10d-48e5db8bf18d

but unlike in case of blobs where the target is a blob file, in our case it is a folder under which we collect manifests as identified by the corresponding zarr checksum: e.g.

128/4a1/1284a14f-fe4f-4dc3-b10d-48e5db8bf18d/6ddc4625befef8d6f9796835648162be-509--710206390.json

which provides a full listing for the zarr, including the ETags per file, and sizes. But for the zarr access, I think only versionid is needed, so we have also a

128/4a1/1284a14f-fe4f-4dc3-b10d-48e5db8bf18d/6ddc4625befef8d6f9796835648162be-509--710206390.versionid.json

which has only versionId per each sub path.

Internally both files follow the same schema, and which fields are listed are listed in the header in the field fields which is either a list when multiple or a string when single value (saving space!), e.g.

yoh@typhon:~/proj/dandi/zarr-manifests$ head -n 25 zarr-manifests-v2-sorted/128/4a1/1284a14f-fe4f-4dc3-b10d-48e5db8bf18d/6ddc4625befef8d6f9796835648162be-509--710206390.json 
{
 "schemaVersion": 2,
 "fields": ["versionId","lastModified","size","ETag"],
 "statistics": {
  "entries": 509,
  "depth": 5,
  "totalSize": 710206390,
  "lastModified": "2022-06-27T23:09:39+00:00",
  "zarrChecksum": "6ddc4625befef8d6f9796835648162be-509--710206390"
 },
 "entries": {
  ".zattrs": ["VwOSu7IVLAQcQHcqOesmlrEDm2sL_Tfs","2022-06-27T23:07:47+00:00",8312,"cb32b88f6488d55818aba94746bcc19a"],
  ".zgroup": ["7obAY5BUNOdI1Uch3RoI4oHuGXhW4h0R","2022-06-27T23:07:47+00:00",24,"e20297935e73dd0154104d4ea53040ab"],
  ".zmetadata": ["Vfe0W0v4zkydmzyXkUMjm2Xr7.rIvfZQ","2022-06-27T23:07:47+00:00",15191,"4f505878fbb943a9793516cf084e07ad"],
  "0": {
   ".zarray": ["Ou6TnKwWPmEJrL.0utCWLPxgfr_lA0I1","2022-06-27T23:07:48+00:00",446,"5477ec3da352681e5ba6f6ea550ef740"],
   "0": {
    "0": {
     "13": {
      "8": {
       "100": ["lqNZ6OQ6lKd2QRW8ekWOiVfdZhiicWsh","2022-06-27T23:09:11+00:00",1793451,"7b5af4c6c28047c83dd86e4814bc0272"],
       "101": ["_i9cZBerb4mB9D8IFbPHo8nrefWcbq0p","2022-06-27T23:09:28+00:00",1799564,"50b6cfb69609319da9bf900a21d5f25c"],
       "103": ["DeraqBNPhVssSggCCTPjmgbu6XxOOupZ","2022-06-27T23:09:22+00:00",1811922,"c88b90dd6d4995fbb32b9a131d1b4ba0"],
       "104": ["J6ejQa4K9cEAjNbkKkZip8.zdK5XFI37","2022-06-27T23:09:20+00:00",1819642,"18d9f3602cb03bfaee7a6f4845db4f90"],
       "106": ["2yX_EGgu8ZxYExf.V.c_kc.jf4ijT8An","2022-06-27T23:09:33+00:00",1837884,"3006f1300734e195b505692e9c301235"],

for the full one so we have everything desired e.g. for webdav to provide listing and access, and compute/validate the zarrChecksum. And the .versionid.json one looks like

yoh@typhon:~/proj/dandi/zarr-manifests$ head -n 25 zarr-manifests-v2-sorted/128/4a1/1284a14f-fe4f-4dc3-b10d-48e5db8bf18d/6ddc4625befef8d6f9796835648162be-509--710206390.versionid.json 
{
 "schemaVersion": 2,
 "fields": "versionId",
 "statistics": {
  "entries": 509,
  "depth": 5,
  "totalSize": 710206390,
  "lastModified": "2022-06-27T23:09:39+00:00",
  "zarrChecksum": "6ddc4625befef8d6f9796835648162be-509--710206390"
 },
 "entries": {
  ".zattrs": "VwOSu7IVLAQcQHcqOesmlrEDm2sL_Tfs",
  ".zgroup": "7obAY5BUNOdI1Uch3RoI4oHuGXhW4h0R",
  ".zmetadata": "Vfe0W0v4zkydmzyXkUMjm2Xr7.rIvfZQ",
  "0": {
   ".zarray": "Ou6TnKwWPmEJrL.0utCWLPxgfr_lA0I1",
   "0": {
    "0": {
     "13": {
      "8": {
       "100": "lqNZ6OQ6lKd2QRW8ekWOiVfdZhiicWsh",
       "101": "_i9cZBerb4mB9D8IFbPHo8nrefWcbq0p",
       "103": "DeraqBNPhVssSggCCTPjmgbu6XxOOupZ",
       "104": "J6ejQa4K9cEAjNbkKkZip8.zdK5XFI37",
       "106": "2yX_EGgu8ZxYExf.V.c_kc.jf4ijT8An",

which results in over twice smaller file but capable only to provide access to current version of zarr.

About

Experimentation toward producing zarr manifests

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published