[emmanuelmathot]

First contribution toward the decision framework (#183). Opened against the feat/decision-framework feature

---

The spine

flowchart TD
    A([Data to publish or migrate]) --> B{Existing data?}

    B -- No --> C{Archival-format<br/>mandate?}
    C -- Yes --> C1[Optimize the mandated format<br/>+ publish a cloud-optimized copy]
    C -- No --> T{{What shape are the data?}}

    B -- Yes --> D{Can you<br/>reprocess?}
    D -- Yes --> T
    D -- No --> P{Can you repackage<br/>the delivered files?}
    P -- Yes --> P1[Format free,<br/>content fixed as delivered]
    P1 --> T
    P -- No --> E{Virtualizable?}
    E -- Yes --> E1[VirtualiZarr / Kerchunk / Icechunk<br/>limited, better than nothing]
    E -- No --> E2[Not yet possible<br/>say what is missing]

    T --> T1(2D raster<br/>COG, Zarr, PMTiles)
    T --> T2(N-D cubes<br/>GeoZarr, COG, CO-HDF5)
    T --> T3(Point cloud<br/>COPC, GeoParquet)
    T --> T4(Vector / tabular<br/>GeoParquet, FlatGeobuf, PMTiles)

    T1 --> R([Refine: intent, object-size lever,<br/>maturity, interpretation])
    T2 --> R
    T3 --> R
    T4 --> R

    classDef entry   fill:#e7eefb,stroke:#3f6db3,stroke-width:2px,color:#152744
    classDef ask     fill:#fff6dc,stroke:#ad7c00,stroke-width:2px,color:#4a3600
    classDef hinge   fill:#dfeffa,stroke:#2a7fb8,stroke-width:3px,color:#0e3247
    classDef family  fill:#e4f4ea,stroke:#39875f,stroke-width:2px,color:#12301e
    classDef partial fill:#fdeadb,stroke:#c06520,stroke-width:2px,color:#48260a
    classDef refine  fill:#ece6f7,stroke:#7355a6,stroke-width:2px,color:#291d44

    class A entry
    class B,C,D,E,P ask
    class T hinge
    class T1,T2,T3,T4 family
    class C1,P1,E1,E2 partial
    class R refine

Loading

Refs #183

[emmanuelmathot]

This first cut does not include the "quantitative harness" enabled by https://github.com/developmentseed/cng-formats-benchmark but mention the possible extension.

[gadomski]

I did a run-through of the first half of index.qmd; I'll try to come back for the rest later when I get another free time window. I think the high level flowchart and framework feels about correct, but we need to be sure to explain (or link to definition) of terms before we use them — as an "insider" I still wondered what some acronyms or phrases meant.

[emmanuelmathot]

Thank you @gadomski for the review. I tried to integrate and improve according to your comments and suggestions. I left some opened, feel free to close them or comment further.
I re-request a review because I made a lot of changes. Thanks.

[wildintellect]

For Zarr I think most of this holds true. However for some of the other formats, there might be a 3rd item in this list, which is to actually split an existing file into separate files because there would be too many chunks/shards in one file the index would be huge. This might be a for later discussion.

[wildintellect]

I'm thinking about how we somewhat manually design geoparquet partitions (across files), Zarr already does this inherently. There is a a practical limit though, you wouldn't want a single COG for the whole world at 1m resolution, even if it is chunked nicely, that file would be hard to work with on a management side.

[abarciauskas-bgse]

I may be missing something but here and elsewhere it seems to be suggested that an internal COG tile could be its own object. I don't think that is the case? I modified wording above "how bytes are grouped into data chunks and chunks grouped into objects" but when we have a COG that is already too large wouldn't the solution have to be to actually break the COG into smaller regions or separate bands?

[emmanuelmathot]

Thx. You are right, and this is an error. For COG, COPC, and single-file GeoParquet, the file is the object. Internal tiles and row groups are byte ranges inside that one object. They do not change the object size at all. Zarr is the only format where the two coincide, because there a shard is an object.

The page also contradicted itself. The "size and shape are two different questions" paragraph below already made this distinction correctly. The table will have two columns, "what sets the object size" and "what sets the read granularity inside the object", and a sentence will state that only Zarr connects them. I will also correct the "products that are already large" bullet, which told you to size the tile or the row group.

[wildintellect]

I'm fairly happy with this as the first version. Will let the others chime in if there are still any blockers for them.

[abarciauskas-bgse]

some only charge you as if your objects were larger

can you explain this a bit more @emmanuelmathot ?

[emmanuelmathot]

Indeed, too vague to be useful. I meant a minimum billable object size. Several S3 storage classes charge for small objects as if they were larger. Standard-IA, One Zone-IA, and Glacier Instant Retrieval each charge a minimum of 128 KB for one object. A 10 KB object then costs the same as a 128 KB object. Glacier Flexible Retrieval and Deep Archive add 40 KB of metadata for each object.

The write does not fail, so you do not see this cost until the invoice. The text now gives this example and links to the AWS storage class table.

Will precise in the following paragraph

[abarciauskas-bgse]

I don't understand what "side metadata" means here. I think what you are saying is that these encodings are baked into the Zarr standard but not these other standards. An example may help.

[wildintellect]

external files, aka sidecar files, somecog.xml

[abarciauskas-bgse]

can we sidecar metadata files or add this term the terminology section?

[emmanuelmathot]

The term is jargon. One small correction on "sidecar files": for a COG with scaled integers, the scale and the offset are usually inside the TIFF, in a GDAL metadata tag. They are not always in a separate file. The important difference is not the location. It is whether the data model of the format makes the reader apply the value.

The term appears only twice, so I removed it instead of adding it to the terminology table. The bullet will give the example you ask for.

[abarciauskas-bgse]

I find this section a bit confusing (why are we talking about bands derived from one another?), perhaps we can discuss it and take a shot at making it more clear

[wildintellect]

oh we can probably clarify that, this is probably adding too much detail about how a band is captured on some satellites. The key point is that some products have bands at different resolutions.

[emmanuelmathot]

This section answers @wildintellect's prevvious comment. An earlier version said mixed resolution was out of scope. We corrected that: the two cases combine into one pyramid when the step between levels is not a fixed factor. Sentinel-2 is the example, and the Zarr multiscales convention uses that same layout as its own reference example.

fair poiny though. Let's decide what we want here:

1. Keep the content, and fix the confusing sentence.
2. Keep a short version here, and move the Sentinel-2 detail to the Zarr page or to the case study.

Adding this point to the call. Leaving this open.

[abarciauskas-bgse]

Pete: "scale factors between levels are not required to be uniform", however if you need to combine bands or variables they need to be the same resolution to avoid resampling

[abarciauskas-bgse]

todo: verify that COGs allow for arbitrary overview resolutions

[gadomski]

https://gdal.org/en/stable/programs/gdal_raster_overview_refresh.html#cmdoption-gdal_raster_overview_refresh-levels 👈🏼 looks like a good place to start to proof this out.

[abarciauskas-bgse]

I wonder if we can simplify this section with just a few sentences and pointing readers to the cng-formats-benchmark repo.

[emmanuelmathot]

Agreed. The section is probably too long here. I will reduce it for this release. However, I would like to defer a future work for a potential dedicated page to describe how to make a quantitative assessement using the benchmark and how to integrate in the decision framework. Followup issue opened here

[abarciauskas-bgse]

I don't understand what "handoff" is being referred to here

[wildintellect]

The framework runs in two stages, and the topology question marks the hand-off between them.

This made more sense to me before, will think about how to be more clear.

[emmanuelmathot]

I introduced "hand-off" to replace "hinge", because I thought it was the clearer word but seems not. I will describe the transition literally.

[abarciauskas-bgse]

This feels out of context and why its a clear anti-pattern is not really explained, so we should probably just remove this sentence.

[emmanuelmathot]

You are right that the sentence was not very clear. I would prefer to fix it than to delete it, because the finding is real and it connects to the object-size lever.

[abarciauskas-bgse]

This is a bit hard to digest without knowing what passes are and more about the data, but maybe that is unavoidable in order to provide a succinct case study. Are there multiple passes per shapefile or is it one per shapefile? Does each pass constitute one Geoparquet row or many?

[emmanuelmathot]

Fair point. One pass is one overflight, and the mission delivers one shapefile per pass. But LakeSP Prior reports the whole prior lake database on every pass, not only the lakes that pass observed, so about 78 % of the rows describe a lake with no observation and carry a NULL geometry. Many rows per pass, and most of them empty. The GeoParquet output keeps the same rows. That fact also drives the aggregation lever: concatenating passes stacks these row sets, and the row groups are sized after that.

I had this wrong in the text and will correct it

[abarciauskas-bgse]

We should add an actual example from NASA here, we have many now.

[abarciauskas-bgse]

How do these thresholds work -- is the mean object size per bucket or some other grouping unit? And you don't have a choice, if you have mean objects of 32 MB or smaller, all those objects are in warm storage and if you have mean object size of greater than 32 MB its in cold storage? Or just greater than 100MB? Then what happens inbetween 32 and 100 MB?

[emmanuelmathot]

Good question. I realise our wording cause the confusion. "Sorts data into tiers" makes the process sound automatic and it is not the case.

The Datalake is an in-house S3 service with a lifecycle policy that moves data between disk and tape. Each tier has a recommended minimum mean object size: about 32 MB for disk, and about 100 MB for tape. The check is mechanical. The system divides the total volume by the object count, then compares the result to the floor for the tier.

• A larger mean object size makes your data eligible for more tiers. It never moves your data to a colder tier.
• Between 32 MB and 100 MB, the data is suitable for disk but not for tape.
  Below 32 MB is the failure case. The penalty is not the tier. Too many small objects fill the object index table before the disk capacity is full. The Datalake then cannot supply its pooled volume to any project, even when much disk space remains.
• That last point is the real motivation, and it is much stronger than "tiers", so I will rewrite the paragraph around it. In CNES, floors are advisory, not enforced.

[abarciauskas-bgse]

We haven't solidly defined products. This only makes sense when there is an existing "product" we are converting from, which we've already stated isn't the case. I think this reads fine without these bullets at all, but if we decide to keep them we should make clear these statements only make sense in the scenario of an existing product we are converting.