From 73c87cd7f05def5e5da476c4feca2a142457c41e Mon Sep 17 00:00:00 2001 From: Emmanuel Mathot Date: Mon, 3 Aug 2026 17:26:59 +0200 Subject: [PATCH 01/59] Add format-selection decision framework page and case studies Adds a two-page decision framework that takes a reader from "I have geospatial data to publish or migrate" to a short list of cloud-optimized format candidates and the layout choices that make each one work. decision-framework/index.qmd - Preconditions P1 (partial/parallel reads over HTTP) and P2 (no opaque bundles), written on top of the guide's existing cloud-optimized format pattern and glossary rather than as a parallel definition. - The spine: a control fork (existing data, reprocessable) then a topology gate that opens a candidate set, with a Mermaid flowchart. - Refinements: intent, the object-size / grouping lever, ecosystem maturity, and interpretation encoding. - Cross-cutting notes: metadata and cataloging (STAC recommended, not prescribed), within-format depth, and dead ends framed as "unlock this". - A short optional section on making the choice quantitative, linking out rather than reproducing the full spec. decision-framework/case-studies.qmd - Six products run through the spine, five backed by measurements, leading with the measured outcome and the recurring lesson that cloud-optimized is rarely smaller and the gap is usually the codec. Also registers both pages in _quarto.yml (placement flagged for review), adds them to the Welcome table of contents, and credits CNES and the Copernicus / NASA source missions in _thankyous.qmd. Refs #183 Co-Authored-By: Claude Opus 5 --- CITATION.cff | 2 + _quarto.yml | 9 ++ _thankyous.qmd | 4 +- decision-framework/case-studies.qmd | 131 ++++++++++++++++++ decision-framework/index.qmd | 206 ++++++++++++++++++++++++++++ index.qmd | 7 +- 6 files changed, 356 insertions(+), 3 deletions(-) create mode 100644 decision-framework/case-studies.qmd create mode 100644 decision-framework/index.qmd diff --git a/CITATION.cff b/CITATION.cff index a8df392f..8de5cd6d 100644 --- a/CITATION.cff +++ b/CITATION.cff @@ -17,6 +17,8 @@ authors: family-names: "Barron" - given-names: "Zachary" family-names: "Deziel" +- given-names: "Emmanuel" + family-names: "Mathot" - given-names: "Max" family-names: "Jones" - given-names: "Brianna" diff --git a/_quarto.yml b/_quarto.yml index 81eb6654..3200d1f1 100644 --- a/_quarto.yml +++ b/_quarto.yml @@ -31,6 +31,15 @@ website: text: Welcome - href: overview.qmd text: Overview Slides + # NOTE for @abarciauskas-bgse: proposed nav placement only. + # Placed before "Formats" so a reader can pick a format family before + # diving into a format page. Final position depends on the Track-3 site + # redesign - please confirm or move (and rename the path if needed). + - section: Choosing a Format + href: decision-framework/index.qmd + contents: + - decision-framework/index.qmd + - decision-framework/case-studies.qmd - section: Formats contents: - section: Cloud Optimized GeoTIFFs (COG) diff --git a/_thankyous.qmd b/_thankyous.qmd index c2b17772..47dce6f1 100644 --- a/_thankyous.qmd +++ b/_thankyous.qmd @@ -10,4 +10,6 @@ This guide has been made possible through the support of: [![](./images/thankyous-devseed-logo.png){fig-alt="Development Seed Logo" fig-align="left" width=40%}](https://developmentseed.org) [![](./images/thankyous-cng-logo.png){fig-alt="Cloud-Native Geospatial Forum" fig-align="left" width=40%}](https://cloudnativegeo.org) -::: \ No newline at end of file +::: + +The [Format Selection Decision Framework](./decision-framework/index.qmd) and its case studies were contributed from a study for the [Centre National d'Études Spatiales (CNES)](https://cnes.fr/), and use public [Copernicus](https://www.copernicus.eu/) and [NASA](https://www.earthdata.nasa.gov/) mission data. \ No newline at end of file diff --git a/decision-framework/case-studies.qmd b/decision-framework/case-studies.qmd new file mode 100644 index 00000000..d812ccf5 --- /dev/null +++ b/decision-framework/case-studies.qmd @@ -0,0 +1,131 @@ +--- +title: "Decision Framework Case Studies" +subtitle: "Six products run through the framework, five of them backed by measurements" +--- + +Each case study below runs a real product through the [decision framework](index.qmd). It shows where the product lands and why. Five of the six are backed by **measured benchmark results**. + +Read each case in two parts: + +1. **The path through the spine**: the control fork, the topology gate, then intent, the object-size lever, and maturity. +2. **What the measurement showed.** + +The examples use public Copernicus and NASA missions. The CNES Datalake appears as one worked example of a tiered object store. It is not the only kind of backend, and its tier numbers are not a general rule. + +::: {.callout-important} +## The recurring lesson + +**Going cloud-optimized is rarely smaller than the source you started from.** + +What it buys you is partial access, a spatial index, or a level of detail. Where two cloud-optimized candidates *do* differ in size, the cause is usually the **codec**, not the format. + +A second, related warning: **compare like for like on content.** More than one surprising number in this study turned out to be a comparison against a file that held fewer variables. +::: + +::: {.callout-note} +The numbers below are first measurements on a representative unit per mission, meaning one scene, one tile, or a few granules. They are illustrative, not a published benchmark. Two of the six cases have re-runs pending, and each one says so. +::: + +## 1. Per-scene optical raster: COG against a Zarr time-stack + +**Product:** Sentinel-2 L2A. **Topology:** 2D raster per scene. **Candidate set:** `{COG, GeoZarr}`. + +**The path.** Existing data, and the producer operates the processing chain, so this is a free choice. The topology opens `{COG, GeoZarr}` and the intent decides between them. Per-scene **distribution and visualization** favors COG. **Time-series analysis** favors a GeoZarr cube. The grouping lever is COG internal tiling, or a scene-size Zarr shard. + +**Measured, one scene.** Both layouts cleared the warm tier. COG came to 2625.7 MiB in 28 objects. A scene-size GeoZarr shard came to 1975.6 MiB in 26 objects, about 25 % smaller. That gap is the **codec**, Deflate against ZSTD, not the format. Against the 2023.7 MiB delivered `.zip`, the GeoZarr store is about source size (−2 %) and the COG is larger (+30 %), because the COG carries overviews. + +A **too-small 2048² shard** split the same bytes into 843 objects that fell below the tier. This is exactly the object-grouping anti-pattern the framework warns about. + +**Interpretation encoding is a real differentiator here.** The source reflectance is stored as scaled integers, and the nodata fill is left undeclared. COG puts the scale, the offset, and the nodata in side metadata that readers honor unevenly. Zarr v3 can carry all three as first-class array properties, using the `scale_offset` codec and a declared fill value, so every reader recovers the same physical value. + +**Where it lands.** COG per scene for distribution and visualization. A scene-size GeoZarr shard for temporal analysis. The associated masking product, which is written as many small GeoTIFFs, is a clear anti-pattern to fix at the source. + +## 2. Raster time-cube: Zarr v3, with COG as the alternative + +**Product:** Sentinel-1 RTC. **Topology:** 2D raster over time. **Candidate set:** `{COG, GeoZarr}`. + +**The path.** Existing data, reprocessable, so a free choice. A year of RTC over a region is naturally a `time × y × x × pol` cube. On a Zarr cube, a time-series slice or an area-of-interest subset is one range-addressable query. Across thousands of per-timestamp COGs, the same request becomes a large fan-out. Sharding reconciles fine-grained chunk access with the backend's preference for large objects. + +**Measured, one tile and confirmed at area scale.** COG came to 728 MiB per tile and a scene-size GeoZarr shard to 545 MiB. Both cleared the cold tier. GeoZarr is again about 25 % smaller, and again this is the codec: Deflate at 3.70x against ZSTD at 4.98x. The result held over a 16-tile block, at 16498 MiB for COG and 12344 MiB for GeoZarr. Against the 496 MiB delivered `.zip` both are larger, COG by 47 % and GeoZarr by 10 %. Cloud-optimized costs storage here, and buys the range access that the opaque zip cannot give at all. + +::: {.callout-warning} +## A benchmark caveat, not a format verdict + +The GeoZarr store read and displayed much slower than the COG, but it was written **with no multiscale overview pyramid**. A store without multiscales is not a faithful GeoZarr, and the missing pyramid is the reason a tile server has to downsample full-resolution chunks on request. The slow read is also in large part a reader effect, from the library used rather than the format. + +Both issues are being fixed for the final campaign, so that display is compared on equal terms. Treat the timing numbers from this arm as provisional. +::: + +## 3. Point cloud: netCDF to COPC + +**Product:** SWOT L2_HR Pixel Cloud. **Topology:** point cloud. **Candidate set:** `{COPC, GeoParquet}`. + +**The path.** Existing data, reprocessable, so a free choice. The point-cloud topology gives COPC, which is a clustered octree inside a LAZ file. The netCDF source has no spatial index and no level of detail, so reading a small geographic subset means scanning the whole granule. + +**Measured, content-complete, three granules.** A COPC carrying all per-point variables was about 16 % **smaller** than the source netCDF. This is the one clear size win among the cases, and it comes from the codec: LASzip at 3.72x against netCDF Deflate at 2.56x. Each granule cleared the tier on its own. + +The real gain is what the netCDF lacks: + +- A **spatial index**. A bounding-box query reads only the overlapping octree nodes. +- A **level of detail**. Octree levels 0 to 11 are the point-cloud analogue of COG overviews. + +::: {.callout-caution} +An earlier measurement on this product looked like a 97 % size saving. It had dropped most of the per-point variables. Size comparisons must be like for like on content. +::: + +**Where it lands.** COPC. The cloud-optimized value is partial access plus level of detail, with a modest size win on top. GeoParquet is the alternative where columnar analytics dominate. + +## 4. Vector and tabular: shapefile to GeoParquet + +**Product:** SWOT L2_HR Lake Single Pass. **Topology:** vector and tabular. **Candidate set:** `{GeoParquet, FlatGeobuf}`. + +**The path.** Existing data, reprocessable, so a free choice. This is a fixed-schema tabular product, with a lake id, a reach id, water-surface elevation, area, and more. That is the native fit for GeoParquet, with its columns, row groups, and spatial partitioning. A sparse Zarr store would add complexity with no gain. + +**Measured, two passes.** GeoParquet came to 17.8 MiB per pass, about 14 % **larger** than the 31.1 MiB zipped shapefile. This surprised reviewers, who expected a binary columnar format to be smaller. The likely cause is the default row-group codec, and it is under investigation. + +Each pass sits below the warm-tier floor. The lever is therefore **aggregation across passes**: concatenate first, then size the row groups. A one-for-one transcode of each pass would leave the object-size problem in place. The gain over the shapefile is the bounding-box row-group index, not the size. + +**Where it lands.** GeoParquet, with pass-level aggregation. FlatGeobuf is the single-file streaming alternative. + +## 5. Small N-D grid: the aggregation regime + +**Product:** SWOT L2_HR Raster. **Topology:** N-D grid. **Candidate set:** `{GeoZarr, COG}`. + +**The path.** Existing data, reprocessable, so a free choice. A 100 m raster granule covers a small tile, about 160 km across. + +**Measured, single variable, so not yet like for like.** On the primary water-surface-elevation variable, COG came to 4.3 MiB per granule and a GeoZarr shard to 0.8 MiB on average, because sharding fragments at this size. Both sit far below the warm-tier floor, so the lever is **aggregation**, and at this small size **COG is the cleaner carrier**: one tiled object that also holds overviews. An all-variable re-run is needed before quoting a size, for the same reason as case 3. + +The natural aggregation unit is a spatial mosaic, per grid zone and per orbit cycle. That turns a wall of tiny granules into one range-addressable object. + +**Where it lands.** COG for the small raster, plus collection-level aggregation. GeoZarr is the alternative to compare once the pyramid and all variables are in place. + +This case is the counterpart to cases 1 and 2. There, one large band already clears the tier, so you group *within* a product. Here the products are small, so you group *above* a single product. + +## 6. Existing archive to a virtual store + +**Topology:** any. **Candidate set:** a virtual Zarr store. + +This case has no benchmark. It illustrates the "existing data, cannot reprocess" branch of the control fork. + +An immutable archive, for example HDF5 or netCDF that you cannot reprocess, can be surfaced as a virtual Zarr store with [VirtualiZarr, Kerchunk, or Icechunk](../zarr/virtual-zarr.qmd). This gives range-based access without rewriting the data. + +The result is more limited than a native store. Chunk sizes are whatever the source chose, you cannot add overviews, and you inherit the source's codec. It is still much better than whole-file download, and it is often the only option available. + +This case is also where the ["unlock this"](index.qmd#dead-ends-are-unlock-this) framing matters. If the source cannot even be virtualized, name what is missing rather than declaring the case impossible. + +## Summary + +| Case | Topology | Lands on | Size against source | Main gain | +| --- | --- | --- | --- | --- | +| 1. Optical scene | 2D raster | COG, or GeoZarr for cubes | COG +30 %, GeoZarr −2 % | Partial access, encoding | +| 2. SAR time-cube | Raster over time | GeoZarr v3 sharded | COG +47 %, GeoZarr +10 % | One query instead of a fan-out | +| 3. Pixel cloud | Point cloud | COPC | −16 % | Spatial index, level of detail | +| 4. Lake vector | Vector, tabular | GeoParquet, aggregated | +14 % | Row-group bbox index | +| 5. Small raster grid | N-D grid | COG, plus aggregation | Re-run pending | Fewer, larger objects | +| 6. Frozen archive | Any | Virtual Zarr | No rewrite | Range access with no reprocessing | + +Read the size column together with the recurring lesson at the top of this page. Only one case got smaller, and that case got smaller because of its codec. + +## Acknowledgements + +These case studies come from a study for the [Centre National d'Études Spatiales (CNES)](https://cnes.fr/), carried out by [Development Seed](https://developmentseed.org/). They use public Copernicus and NASA mission data. See [Our Supporters](../contributing.qmd#thank-you-to-our-supporters). diff --git a/decision-framework/index.qmd b/decision-framework/index.qmd new file mode 100644 index 00000000..c09a7aec --- /dev/null +++ b/decision-framework/index.qmd @@ -0,0 +1,206 @@ +--- +title: "Format Selection Decision Framework" +subtitle: "How to go from data you must publish to a short list of cloud-optimized format candidates" +--- + +## What this framework is + +This framework helps you go from *"I have geospatial data to publish or migrate"* to a **short list of cloud-optimized format candidates**, and to the **layout choices** that make each candidate work. + +It is a guide, not an oracle: + +- The **shape of your data** narrows the field. +- The **intended use** and a few practical levers rank the remaining options. +- **You keep the final call.** + +The framework is provider-neutral. Where a concrete number helps, such as an object-size policy or a benchmark result, the number is given as a worked example and flagged as an example, not as a rule. + +This is a first, deliberately simple version. It aligns on the questions to ask and the order to ask them in. Follow the links to a format page when you need more detail. + +::: {.callout-note} +## What this framework is not + +- It is not a compliance checklist. No format is disqualified by a single answer. +- It does not prescribe a catalog or a metadata standard. See [Metadata and cataloging](#metadata-and-cataloging). +- It does not replace a benchmark. When two candidates are close, measure them. +::: + +## Preconditions + +This guide already defines what makes a format cloud-optimized. The Welcome page states it as a four-part [cloud-optimized format pattern](../index.qmd#the-opportunity), and the glossary defines [cloud-optimized](../glossary.qmd#cloud-optimized) as the ability to read a meaningful *part* of a file without downloading *all* of it. + +The framework does not restate that definition. It assumes it. Two consequences are worth naming here, because they decide whether a candidate belongs in the framework at all. They are entry conditions, not questions in the flowchart. + +**P1. The format must support partial and parallel reads over HTTP.** [Random access](../glossary.qmd#random-access) through [HTTP range requests](../glossary.qmd#http-range-request) is what makes a format usable from [object storage](../glossary.qmd#object-storage-cloud-storage). It depends on the rest of the pattern: [metadata](../glossary.qmd#metadata) in a consistent place that a reader can load in a single read, and then addressable [chunks](../glossary.qmd#chunk), internal tiling, or both. A reader that cannot locate the bytes it needs from the metadata alone cannot fetch only those bytes. + +**P2. No opaque bundles.** An [archive format](../glossary.qmd#archive-format) is not disqualified just for being an archive. [PMTiles](../pmtiles/intro.qmd) is an archive and it is cloud-optimized, because its internal directory makes each tile addressable. What disqualifies a candidate is an archive with no addressable internal layout, or a layer of [external compression](../glossary.qmd#external-compression) wrapped around data that was already cloud-optimized. A zipped Zarr store, a monolithic LAZ file, and a single Parquet file with no row groups all fail this way: the client must fetch the whole object before it can use any part of it. + +P2 is easy to fail by accident, and it is worth checking separately from object size. A bucket can hold large, well-sized objects and still fail P2, because the format inside each object is opaque. Object size and internal structure are two independent axes, and a candidate must satisfy both. + +## The spine + +Two questions come first. They apply to all geospatial data. + +1. **Control.** Do you have existing data, and can you reprocess it? This decides whether you can choose freely, whether you must virtualize, or whether you have reached a dead end. +2. **Topology.** What shape is the data? This selects the **format family**. It opens a candidate set. It does not pick one format. + +Three refinements then rank the candidates: **intent**, the **object-size and grouping lever**, and **ecosystem maturity**. A fourth axis, **interpretation encoding**, is worth a separate look, because the formats differ a lot on it. + +```{mermaid} +flowchart TD + A[Data to publish or migrate] --> B{Existing data?} + B -- No --> C{Archival-format mandate?} + C -- Yes --> C1[Cloud-optimized HDF5 guidance + pick a CN access format] + C -- No --> T[Topology gate] + B -- Yes --> D{Can you reprocess?} + D -- Yes --> T + D -- No --> E{Optimizable / virtualizable?} + E -- Yes --> E1[VirtualiZarr / Kerchunk / Icechunk: limited, better than nothing] + E -- No --> E2[Dead end: not yet funded, unlock this] + T --> T1[2D raster, COG] + T --> T2[N-D cube, Zarr v3 / GeoZarr] + T --> T3[Point cloud, COPC] + T --> T4[Vector / tabular, GeoParquet] + T1 --> R[Refine: intent, object-size lever, maturity, interpretation] + T2 --> R + T3 --> R + T4 --> R +``` + +### The control fork + +**No existing data.** You are designing a new product, so you can choose freely. Go to the topology gate. If a mandate requires you to keep an archival format such as HDF5 or netCDF, you have two jobs: apply the [cloud-optimized HDF/NetCDF](../cloud-optimized-netcdf4-hdf5/index.qmd) guidance to the archive, and choose a cloud-optimized access format for the data you publish. + +**Existing data you can reprocess.** You control the production chain, so you can also choose freely. Go to the topology gate. + +**Existing data you cannot reprocess.** You can often still publish a virtual view over the bytes you have. [Virtual Zarr](../zarr/virtual-zarr.qmd), with VirtualiZarr, Kerchunk, or Icechunk, builds an index that points into the original files. This gives range-based access without rewriting the data. The result is more limited than a native store, but it is much better than nothing. + +**Neither.** Some sources cannot be optimized or virtualized today. Examples are HDF4, encrypted files, and a single large uncompressed blob. See [Dead ends are "unlock this"](#dead-ends-are-unlock-this). + +### The topology gate + +The shape of the data opens a candidate set: + +| Topology | Primary candidate | Common alternatives | +| --- | --- | --- | +| 2D raster, per scene or per tile | [COG](../cloud-optimized-geotiffs/intro.qmd) | Zarr, if the product is really a cube | +| N-D cube, for example `time × y × x × band` | [Zarr v3 / GeoZarr](../zarr/intro.qmd) | COG per time step, if slices are read alone | +| Point cloud | [COPC](../copc/index.qmd) | [GeoParquet](../geoparquet/index.qmd), if columnar analytics dominate | +| Vector or tabular | [GeoParquet](../geoparquet/index.qmd) | [FlatGeobuf](../flatgeobuf/intro.qmd) for streaming, [PMTiles](../pmtiles/intro.qmd) for tiled delivery | + +The gate is about topology, not about the format the data arrives in. A netCDF granule that holds a point cloud belongs on the point-cloud branch. See the [format table](../index.qmd#data-type-to-traditional-to-cloud-optimized-geospatial-file-format-table) for how traditional formats map to cloud-optimized ones. + +## Refinements + +Apply these within the candidate set your topology opened. + +### Intent + +What is the data for? Archive and download, processing, visualization, or distribution. Two points matter: + +- **Visualization is a first-class intent**, not a side effect. It has its own layout needs, mainly [overviews](../glossary.qmd#overviews) or a level of detail. Max Jones maintains a sibling visualization decision tree; use it together with this page when your product is display-driven. +- **On-the-fly and batched processing pull the layout in opposite directions.** On-the-fly access rewards small chunks, fine overviews, and cache locality. Batched processing rewards large shards or row groups and a layout that reads well in sequence. + +Most real datasets serve several intents at once. Weigh them in proportion rather than picking one. + +### The object-size and grouping lever + +Every candidate format has a knob that sets how bytes are grouped into objects: + +| Format | Grouping lever | +| --- | --- | +| COG | internal tiling and overviews | +| Zarr v3 | chunks within shards ([sharding](../glossary.qmd#chunk-size)) | +| COPC | octree nodes | +| GeoParquet | row groups, plus partitioning | + +Tune the lever to your **storage backend's object-size policy**. [Object stores](../glossary.qmd#object-storage-cloud-storage) degrade when they must index very many tiny objects, so most backends reward a larger average object. + +One worked example: the CNES Datalake sorts data into tiers by mean object size, with about 32 MB or more for warm disk and about 100 MB or more for cold tape. Those numbers belong to that backend. Your backend will have its own policy, and it may have none. Read the tiers as an instance of a general concern. + +The lever works in both directions, and which direction you need depends on product size: + +- **Products that are already large.** Group *within* one product. Size the shard, the tile, or the row group so it clears the policy. +- **Products that are small.** Sharding within one granule only makes the fragmentation worse. Group *above* the product instead: mosaic or concatenate many granules into one range-addressable object. + +### Ecosystem maturity + +The real question is not a version number. It is: do you need something production-grade that you can count on, or are you willing to be an early adopter? + +| Level | Formats today | +| --- | --- | +| Mature | COG, the reference case | +| Stable | GeoParquet, COPC, FlatGeobuf | +| Standards track, still ratifying | GeoZarr | + +### Interpretation encoding + +Does the format carry the information a reader needs to recover the same physical value, or does it lean on a side catalog? The information is units, the [CRS](../glossary.qmd#coordinate-reference-system-crs), the nodata fill, and any scale and offset transform. + +This axis separates the candidates more than people expect: + +- **Zarr v3 is distinctively strong here.** With the `scale_offset` codec, a declared fill value, CF conventions, and a CRS convention, the transform is a first-class array property. Every reader recovers the same value. +- **COG, GeoParquet, and COPC standardize the CRS** but leave the value transform to side metadata. Readers honor side metadata unevenly. + +If your product stores scaled integers, or if an undeclared fill value would change a computed result, weigh this axis heavily. + +## Cross-cutting considerations + +These are not gates. They apply whatever you choose. + +### Metadata and cataloging + +Metadata that a reader can fetch in a single request is part of [P1](#preconditions), not an extra. Each candidate in the topology gate has its own mechanism for it: Zarr consolidated metadata, COG headers, GeoParquet footers, and the COPC header. What this section adds is the level *above* a single file. + +At a scale of tens of thousands to millions of files, collection-level discovery becomes critical. [STAC](https://stacspec.org/) has an essential role there, and it is part of the cloud-native geospatial ecosystem that we recommend. It is **not a format and not a gate in this framework**, so this page does not prescribe it. Choose a catalog approach that fits your organization. + +### Within-format depth + +Some choices sit below the format and are foundational: + +- Base codecs, such as [ZSTD](../glossary.qmd#zstd) and [Deflate](../glossary.qmd#deflate) +- Data types and scale-offset encoding +- Sharding in Zarr v3 + +These are mature and in scope. The per-format pages cover them. More experimental optimizations, such as bit-rounding and advanced compression tricks, are out of scope here, or should be marked as immature when you use them. + +### Dead ends are "unlock this" + +When a source cannot be optimized or virtualized today, the honest answer is *"not yet implemented"*, not *"impossible"*. The blocker is usually missing tooling or missing funding, and both can change. + +Frame the branch that way, and route the reader back to the options: reprocess if you can get the chain changed, virtualize if a reader exists, or convert on ingest. Say what is missing, so somebody can unlock it. + +## Optional: make it quantitative + +The spine above gets you to a short list. When two or three candidates remain and the choice is close, a transparent score can make the trade-off explicit rather than a matter of taste. This section is **optional**. The framework stands on its own without it. + +The idea is a **weighted mean of criteria**, each expressed as a number from 0 to 1. It is framework-first and metrics-optional: qualitative terms alone already give a ranking, and a benchmark run sharpens the same ranking. + +| Group | Criterion | +| --- | --- | +| Qualitative, no benchmark needed | **use-case fit**: your intent mix, applied to a per-format suitability matrix | +| | **maturity**: the maturity level above, mapped to a utility | +| | **interpretation encoding**: the encoding level above, mapped to a utility | +| Measured, needs a benchmark run | **size**: output volume against the source | +| | **performance**: measured read, write, and display, blended by the intent mix | + +Rank the candidates and read the **margin**, which is the gap to the runner-up. A small margin is an honest signal that a benchmark, or a review of your weights, is what actually decides it. There are **no verdicts and no probability readout**. You get a score, a ranking, and a margin. + +Two points are easy to get wrong: + +- **Object-size and tier fit is reported information, not a scored penalty.** Report whether a layout clears a given tier, the mean and median object size, and a small-file warning **beside** the score, never inside it. A storage backend's policy is a property of that backend, not of the format, so you must weigh it against your own infrastructure. +- **Intent weights are your input.** The suitability matrix is a shared expert reference on each format's strengths, and it is useful on its own. The intent mix and the criterion weights belong to your context. + +The convincing artifact is never a single score. It is a sensitivity sweep that shows the ranking holds across a range of weights. + +## Worked examples + +Six products run through the spine on the [Case Studies](case-studies.qmd) page. Five are backed by measurements. + +One lesson recurs often enough to state up front: **going cloud-optimized is rarely smaller than the source you started from.** What it buys is partial access, a spatial index, or a level of detail. Where two candidates do differ in size, the cause is usually the **codec**, not the format. + +## Acknowledgements + +This framework was developed as part of a study for the [Centre National d'Études Spatiales (CNES)](https://cnes.fr/), carried out by [Development Seed](https://developmentseed.org/). The case studies use public Copernicus and NASA missions, and the measurements come from that study's benchmark work. + +The framework builds on earlier work in this guide supported by [NASA IMPACT](https://impact.earthdata.nasa.gov/). See [Our Supporters](../contributing.qmd#thank-you-to-our-supporters). diff --git a/index.qmd b/index.qmd index 01be62f5..ec88206e 100644 --- a/index.qmd +++ b/index.qmd @@ -64,7 +64,9 @@ Notes: ## Table of Contents 1. [Overview of Formats (slideshow)](./overview.qmd) -2. Formats +2. [Format Selection Decision Framework](./decision-framework/index.qmd) + a. [Case Studies](./decision-framework/case-studies.qmd) +3. Formats a. [Cloud-Optimized GeoTIFFs](./cloud-optimized-geotiffs/intro.qmd) b. [Zarr](./zarr/intro.qmd) c. [Cloud-Optimized NetCDF4/HDF5](./cloud-optimized-netcdf4-hdf5/index.qmd) @@ -72,7 +74,7 @@ Notes: e. [GeoParquet](./geoparquet/index.qmd) f. [FlatGeobuf](./flatgeobuf/intro.qmd) g. [PMTiles](./pmtiles/intro.qmd) -3. [Cookbooks](./cookbooks/index.qmd) +4. [Cookbooks](./cookbooks/index.qmd) ## Running Examples @@ -85,6 +87,7 @@ Most of the data formats covered in this guide have a Jupyter Notebook example t * [Alex Mandel](https://developmentseed.org/team/alex-mandel) * [Kyle Barron](https://github.com/kylebarron) * [Zac Deziel](https://developmentseed.org/team/zac-deziel) +* [Emmanuel Mathot](https://developmentseed.org/team/emmanuel-mathot) * [Overview Slide](./overview.qmd) credits: Vincent Sarago, Chris Holmes, Patrick Quinn, Matt Hanson, Ryan Abernathey ## Questions to Ask When Generating Cloud-Optimized Geospatial Data in Any Format From 7e37124783e4002a8d9d76973476a65169a4b9af Mon Sep 17 00:00:00 2001 From: Emmanuel Mathot Date: Mon, 3 Aug 2026 17:31:40 +0200 Subject: [PATCH 02/59] Link cng-formats-benchmark as the quantitative enabler The optional scoring section named two measured criteria (size and performance) but had no outbound link, so a reader had no way to reproduce a number. Point both pages at the open-source harness the case-study measurements actually came from. - Framework page: a "where the measured numbers come from" paragraph after the criteria table, naming what the harness measures (object-size distribution and tier fitness, write, range-request-aware read, tile display), that datasets are described in config files rather than baked in, and linking the docs and configuration guide. - Case studies: provenance in the measurement caveat, plus a note in the acknowledgements that the harness is maintained outside this guide so its methodological choices stay debatable on their own terms (ADR 0002). Refs #183 Co-Authored-By: Claude Opus 5 --- decision-framework/case-studies.qmd | 4 +++- decision-framework/index.qmd | 4 +++- 2 files changed, 6 insertions(+), 2 deletions(-) diff --git a/decision-framework/case-studies.qmd b/decision-framework/case-studies.qmd index d812ccf5..a9727981 100644 --- a/decision-framework/case-studies.qmd +++ b/decision-framework/case-studies.qmd @@ -24,6 +24,8 @@ A second, related warning: **compare like for like on content.** More than one s ::: {.callout-note} The numbers below are first measurements on a representative unit per mission, meaning one scene, one tile, or a few granules. They are illustrative, not a published benchmark. Two of the six cases have re-runs pending, and each one says so. + +They were produced with [`cng-formats-benchmark`](https://github.com/developmentseed/cng-formats-benchmark), an open-source harness that measures object-size distribution, write, read, and display per format. You can describe your own datasets in its config files and reproduce this shape of comparison on your own data. See [Optional: make it quantitative](index.qmd#optional-make-it-quantitative). ::: ## 1. Per-scene optical raster: COG against a Zarr time-stack @@ -128,4 +130,4 @@ Read the size column together with the recurring lesson at the top of this page. ## Acknowledgements -These case studies come from a study for the [Centre National d'Études Spatiales (CNES)](https://cnes.fr/), carried out by [Development Seed](https://developmentseed.org/). They use public Copernicus and NASA mission data. See [Our Supporters](../contributing.qmd#thank-you-to-our-supporters). +These case studies come from a study for the [Centre National d'Études Spatiales (CNES)](https://cnes.fr/), carried out by [Development Seed](https://developmentseed.org/). They use public Copernicus and NASA mission data. The measurements were produced with [`cng-formats-benchmark`](https://github.com/developmentseed/cng-formats-benchmark), which is maintained separately from this guide so that its methodological choices stay debatable in their own right. See [Our Supporters](../contributing.qmd#thank-you-to-our-supporters). diff --git a/decision-framework/index.qmd b/decision-framework/index.qmd index c09a7aec..0144e81f 100644 --- a/decision-framework/index.qmd +++ b/decision-framework/index.qmd @@ -184,6 +184,8 @@ The idea is a **weighted mean of criteria**, each expressed as a number from 0 t | Measured, needs a benchmark run | **size**: output volume against the source | | | **performance**: measured read, write, and display, blended by the intent mix | +**Where the measured numbers come from.** The two measured criteria need a harness, and reproducibility is the whole point of quoting a number at all. [`cng-formats-benchmark`](https://github.com/developmentseed/cng-formats-benchmark) is the open-source (MIT) harness behind the numbers on the [Case Studies](case-studies.qmd) page. Per format and per dataset it measures the object-size distribution and tier fitness, write throughput, range-request-aware read latency, and tile-display latency. You describe your own datasets in config files, so no provider or dataset is baked into the code, and you run it on your own infrastructure against your own data. See the [documentation](https://developmentseed.github.io/cng-formats-benchmark/) and the [configuration guide](https://developmentseed.github.io/cng-formats-benchmark/configuration/). + Rank the candidates and read the **margin**, which is the gap to the runner-up. A small margin is an honest signal that a benchmark, or a review of your weights, is what actually decides it. There are **no verdicts and no probability readout**. You get a score, a ranking, and a margin. Two points are easy to get wrong: @@ -191,7 +193,7 @@ Two points are easy to get wrong: - **Object-size and tier fit is reported information, not a scored penalty.** Report whether a layout clears a given tier, the mean and median object size, and a small-file warning **beside** the score, never inside it. A storage backend's policy is a property of that backend, not of the format, so you must weigh it against your own infrastructure. - **Intent weights are your input.** The suitability matrix is a shared expert reference on each format's strengths, and it is useful on its own. The intent mix and the criterion weights belong to your context. -The convincing artifact is never a single score. It is a sensitivity sweep that shows the ranking holds across a range of weights. +The convincing artifact is never a single score. It is a sensitivity sweep that shows the ranking holds across a range of weights. That is also the honest reason this section stays short and links out: a score you cannot reproduce on your own data is not worth much, so the harness matters more than any number this page could print. ## Worked examples From cd125999bf2744eaf93266088d4caa90d1a25305 Mon Sep 17 00:00:00 2001 From: Emmanuel Mathot Date: Mon, 3 Aug 2026 17:38:56 +0200 Subject: [PATCH 03/59] Reframe the second size lesson around overviews and indexes The recurring-lesson callout paired the "rarely smaller" headline with a measurement-hygiene warning. Replace it with the mechanism that actually explains the gap: overviews, levels of detail and spatial indexes are structures the source never carried, so they add volume, and that volume is the feature being bought. Anchored to the two cases that show it (the COG carrying overviews, the GeoParquet row-group index). The like-for-like-on-content caveat is dropped from the summary only. It stays inline in cases 3 and 5, where it is concrete and actionable. Also aligns the same paragraph on the framework page. Refs #183 Co-Authored-By: Claude Opus 5 --- decision-framework/case-studies.qmd | 2 +- decision-framework/index.qmd | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/decision-framework/case-studies.qmd b/decision-framework/case-studies.qmd index a9727981..29be1d6a 100644 --- a/decision-framework/case-studies.qmd +++ b/decision-framework/case-studies.qmd @@ -19,7 +19,7 @@ The examples use public Copernicus and NASA missions. The CNES Datalake appears What it buys you is partial access, a spatial index, or a level of detail. Where two cloud-optimized candidates *do* differ in size, the cause is usually the **codec**, not the format. -A second, related warning: **compare like for like on content.** More than one surprising number in this study turned out to be a comparison against a file that held fewer variables. +A second element explains much of the gap: **the extra bytes are the feature.** Overviews, levels of detail, and spatial indexes are structures the source file did not carry at all. They add volume, and they are exactly what buys fast display and cheap subsetting. The COG in case 1 is about 30 % larger than the delivered archive mostly because it carries overviews, and the row-group index in case 4 is the whole point of the conversion. ::: ::: {.callout-note} diff --git a/decision-framework/index.qmd b/decision-framework/index.qmd index 0144e81f..68432cf0 100644 --- a/decision-framework/index.qmd +++ b/decision-framework/index.qmd @@ -199,7 +199,7 @@ The convincing artifact is never a single score. It is a sensitivity sweep that Six products run through the spine on the [Case Studies](case-studies.qmd) page. Five are backed by measurements. -One lesson recurs often enough to state up front: **going cloud-optimized is rarely smaller than the source you started from.** What it buys is partial access, a spatial index, or a level of detail. Where two candidates do differ in size, the cause is usually the **codec**, not the format. +One lesson recurs often enough to state up front: **going cloud-optimized is rarely smaller than the source you started from.** Overviews, levels of detail, and spatial indexes are structures the source did not carry, so they add volume, and that added volume is the feature you are buying. Where two candidates do differ in size, the cause is usually the **codec**, not the format. ## Acknowledgements From 0d3e0d2a69d9792c904bcf178dc18df23316463b Mon Sep 17 00:00:00 2001 From: Emmanuel Mathot Date: Tue, 4 Aug 2026 08:47:16 +0200 Subject: [PATCH 04/59] Expand ecosystem maturity with an explicit scale and the Zarr/GeoZarr split The maturity refinement was a three-row table with no definitions, which made it the weakest of the four axes. Replace it with the substance from the strategic-report state-of-the-art work: - An explicit L1-L5 scale, so the judgement is stated rather than impressionistic, and a per-format table giving specification status, tooling and level. This also separates FlatGeobuf (L3) from GeoParquet and COPC (L4), which the old table lumped together. - The distinction that is easiest to get wrong: Zarr v3 the substrate is dependable (core spec final in 2024, sharding stable in zarr-python 3.x and readable by GDAL and xarray), while GeoZarr the convention layer is still being ratified in an OGC SWG. The L2 belongs to the conventions, not to Zarr. - "Maturity is a warning, not a veto", which reconciles the topology gate offering GeoZarr as the primary N-D cube candidate with that candidate sitting at L2: there is no mature alternative at that topology, so a low level means plan for change, pin versions, and expect the tooling to improve. Points at geozarr.org for current status rather than quoting a ratification date that will age badly. Refs #183 Co-Authored-By: Claude Opus 5 --- decision-framework/index.qmd | 51 +++++++++++++++++++++++++++++------- 1 file changed, 42 insertions(+), 9 deletions(-) diff --git a/decision-framework/index.qmd b/decision-framework/index.qmd index 68432cf0..0603f260 100644 --- a/decision-framework/index.qmd +++ b/decision-framework/index.qmd @@ -98,7 +98,7 @@ Apply these within the candidate set your topology opened. What is the data for? Archive and download, processing, visualization, or distribution. Two points matter: -- **Visualization is a first-class intent**, not a side effect. It has its own layout needs, mainly [overviews](../glossary.qmd#overviews) or a level of detail. Max Jones maintains a sibling visualization decision tree; use it together with this page when your product is display-driven. +- **Visualization is a first-class intent**, not a side effect. It has its own layout needs, mainly [overviews](../glossary.qmd#overviews) or a level of detail. - **On-the-fly and batched processing pull the layout in opposite directions.** On-the-fly access rewards small chunks, fine overviews, and cache locality. Batched processing rewards large shards or row groups and a layout that reads well in sequence. Most real datasets serve several intents at once. Weigh them in proportion rather than picking one. @@ -116,7 +116,7 @@ Every candidate format has a knob that sets how bytes are grouped into objects: Tune the lever to your **storage backend's object-size policy**. [Object stores](../glossary.qmd#object-storage-cloud-storage) degrade when they must index very many tiny objects, so most backends reward a larger average object. -One worked example: the CNES Datalake sorts data into tiers by mean object size, with about 32 MB or more for warm disk and about 100 MB or more for cold tape. Those numbers belong to that backend. Your backend will have its own policy, and it may have none. Read the tiers as an instance of a general concern. +One worked example: the [CNES Datalake](https://cnes.fr/en/projects/centre-de-calcul) sorts data into tiers by mean object size, with about 32 MB or more for warm disk and about 100 MB or more for cold tape. Those numbers belong to that backend. Your backend will have its own policy, and it may have none. Read the tiers as an instance of a general concern. The lever works in both directions, and which direction you need depends on product size: @@ -125,13 +125,46 @@ The lever works in both directions, and which direction you need depends on prod ### Ecosystem maturity -The real question is not a version number. It is: do you need something production-grade that you can count on, or are you willing to be an early adopter? +The real question is not a version number. It is whether you need something production-grade that you can count on, or you are willing to be an early adopter. Maturity is what separates a safe default from a bet. -| Level | Formats today | +To keep that judgement explicit rather than impressionistic, place each candidate on a five-level scale: + +| Level | Meaning | | --- | --- | -| Mature | COG, the reference case | -| Stable | GeoParquet, COPC, FlatGeobuf | -| Standards track, still ratifying | GeoZarr | +| **L5** | De-facto standard, universal tooling, no specification risk | +| **L4** | Established: released spec, broad independent tooling, in production use | +| **L3** | Stable: released spec, multiple implementations, growing tooling | +| **L2** | Emerging: spec stabilizing, reference implementations, early adopters | +| **L1** | Experimental: draft or in-flux spec, few implementations | + +Where the candidates sit today: + +| Format | Specification | Tooling | Level | +| --- | --- | --- | --- | +| [COG](../cloud-optimized-geotiffs/intro.qmd) | De-facto standard, on a GeoTIFF base | GDAL, rio-tiler, TiTiler, every GIS | **L5** | +| [GeoParquet](../geoparquet/index.qmd) | v1.x, stable, on an Apache Parquet base | GDAL, GeoPandas, DuckDB, Arrow | **L4** | +| [COPC](../copc/index.qmd) | v1.0, stable, on a LAZ base | PDAL, untwine, QGIS, copc.js | **L4** | +| [FlatGeobuf](../flatgeobuf/intro.qmd) | Stable, on a flatbuffers base | GDAL and OGR, QGIS, flatgeobuf.js | **L3** | +| [GeoZarr](../zarr/intro.qmd) | Draft, in an OGC Standards Working Group | Emerging, mostly xarray-based stacks | **L2** | + +**COG is the reference case** and the format the others are measured against: a de-facto standard for years, supported by every relevant library and GIS, with no open specification risk. + +**For GeoParquet and COPC, the risk is integration effort, not the specification.** Each inherits a mature base format and adds a spatial index on top. + +#### Zarr v3 and GeoZarr are two different questions + +This distinction is easy to miss, and it decides how much risk you actually take on. + +- **Zarr v3, the substrate, is dependable.** The core specification was finalized in 2024. [Sharding](#the-object-size-and-grouping-lever), the feature that matters most for object storage, is stable in `zarr-python` 3.x and readable by GDAL and xarray. Judged on its own, the Zarr v3 core behaves like an L4. +- **GeoZarr, the convention layer, is still being ratified.** It is what makes a generic [Zarr](../glossary.qmd#zarr) store *geospatial*: CRS, dimension semantics, and multiscale pyramids. It does for Zarr roughly what the GeoTIFF tags do for a plain TIFF. It is a draft in an OGC Standards Working Group, on the standards track but not yet an adopted OGC standard, and the published draft can lag behind active convention work. Check [geozarr.org](https://geozarr.org/) for the current status before you rely on a specific version. + +So the L2 above belongs to the convention layer, not to Zarr. The leading reference implementations today, such as ESA's EOPF Sentinel output, are demonstrators rather than production guarantees. + +#### Maturity is a warning, not a veto + +An L2 candidate can still be the right answer. The N-D cube branch is exactly that case: there is no mature alternative with the same topology, so a low level does not mean "choose something else". It means **plan for change**. Budget for convention churn, pin the versions you depend on, and expect the visualization story in particular to improve as multiscale tooling catches up. + +Read maturity against your own tolerance. A one-off internal product and a mission archive with a twenty-year retention commitment can reasonably land on different levels from the same candidate set. ### Interpretation encoding @@ -139,7 +172,7 @@ Does the format carry the information a reader needs to recover the same physica This axis separates the candidates more than people expect: -- **Zarr v3 is distinctively strong here.** With the `scale_offset` codec, a declared fill value, CF conventions, and a CRS convention, the transform is a first-class array property. Every reader recovers the same value. +- **Zarr v3 is distinctively strong here.** With the `scale_offset` codec, a declared fill value, Zarr conventions, and a CRS convention, the transform is a first-class array property. Every reader recovers the same value. - **COG, GeoParquet, and COPC standardize the CRS** but leave the value transform to side metadata. Readers honor side metadata unevenly. If your product stores scaled integers, or if an undeclared fill value would change a computed result, weigh this axis heavily. @@ -179,7 +212,7 @@ The idea is a **weighted mean of criteria**, each expressed as a number from 0 t | Group | Criterion | | --- | --- | | Qualitative, no benchmark needed | **use-case fit**: your intent mix, applied to a per-format suitability matrix | -| | **maturity**: the maturity level above, mapped to a utility | +| | **maturity**: the L1 to L5 level above, mapped to a utility (`level / 5`) | | | **interpretation encoding**: the encoding level above, mapped to a utility | | Measured, needs a benchmark run | **size**: output volume against the source | | | **performance**: measured read, write, and display, blended by the intent mix | From 69e2a2f8451d7469ec4f61ecbf711029f482b7a4 Mon Sep 17 00:00:00 2001 From: Emmanuel Mathot Date: Tue, 4 Aug 2026 09:33:41 +0200 Subject: [PATCH 05/59] Update .gitignore to exclude Quarto notebook files and enhance clarity in decision framework section on Zarr v3 --- .gitignore | 2 ++ decision-framework/index.qmd | 2 +- 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/.gitignore b/.gitignore index 66f46915..0f3b41e4 100644 --- a/.gitignore +++ b/.gitignore @@ -15,3 +15,5 @@ __pycache__/ .ipynb_checkpoints /.luarc.json + +**/*.quarto_ipynb diff --git a/decision-framework/index.qmd b/decision-framework/index.qmd index 0603f260..493f8303 100644 --- a/decision-framework/index.qmd +++ b/decision-framework/index.qmd @@ -172,7 +172,7 @@ Does the format carry the information a reader needs to recover the same physica This axis separates the candidates more than people expect: -- **Zarr v3 is distinctively strong here.** With the `scale_offset` codec, a declared fill value, Zarr conventions, and a CRS convention, the transform is a first-class array property. Every reader recovers the same value. +- **Zarr v3 makes the information first-class.** With the `scale_offset` codec, a declared fill value, Zarr conventions for projection, CRS and multiscale, the transform is a first-class array property. Every reader recovers the same value. - **COG, GeoParquet, and COPC standardize the CRS** but leave the value transform to side metadata. Readers honor side metadata unevenly. If your product stores scaled integers, or if an undeclared fill value would change a computed result, weigh this axis heavily. From ef5b027d9e8995f70bbb6515252ddf8ff87d6f7b Mon Sep 17 00:00:00 2001 From: Emmanuel Mathot Date: Tue, 4 Aug 2026 10:00:20 +0200 Subject: [PATCH 06/59] Enhance decision framework documentation by clarifying intent categories and updating case study references for improved context and accuracy. --- decision-framework/case-studies.qmd | 40 +++++++++++++++++++---------- decision-framework/index.qmd | 5 ++-- 2 files changed, 30 insertions(+), 15 deletions(-) diff --git a/decision-framework/case-studies.qmd b/decision-framework/case-studies.qmd index 29be1d6a..67d15af5 100644 --- a/decision-framework/case-studies.qmd +++ b/decision-framework/case-studies.qmd @@ -10,7 +10,7 @@ Read each case in two parts: 1. **The path through the spine**: the control fork, the topology gate, then intent, the object-size lever, and maturity. 2. **What the measurement showed.** -The examples use public Copernicus and NASA missions. The CNES Datalake appears as one worked example of a tiered object store. It is not the only kind of backend, and its tier numbers are not a general rule. +The examples use public Copernicus and NASA missions. The [CNES Datalake](https://cnes.fr/en/projects/centre-de-calcul) appears as one worked example of a tiered object store. It is not the only kind of backend, and its tier numbers are not a general rule. ::: {.callout-important} ## The recurring lesson @@ -28,23 +28,31 @@ The numbers below are first measurements on a representative unit per mission, m They were produced with [`cng-formats-benchmark`](https://github.com/developmentseed/cng-formats-benchmark), an open-source harness that measures object-size distribution, write, read, and display per format. You can describe your own datasets in its config files and reproduce this shape of comparison on your own data. See [Optional: make it quantitative](index.qmd#optional-make-it-quantitative). ::: -## 1. Per-scene optical raster: COG against a Zarr time-stack +## 1. Per-scene optical raster: COG vs GeoZarr for visualization and distribution -**Product:** Sentinel-2 L2A. **Topology:** 2D raster per scene. **Candidate set:** `{COG, GeoZarr}`. +**Product:** Sentinel-2 L2A. **Topology:** 2D raster per scene. **Source format:** geoTiff in a `.zip`. **Candidate set:** `{COG, GeoZarr}`. -**The path.** Existing data, and the producer operates the processing chain, so this is a free choice. The topology opens `{COG, GeoZarr}` and the intent decides between them. Per-scene **distribution and visualization** favors COG. **Time-series analysis** favors a GeoZarr cube. The grouping lever is COG internal tiling, or a scene-size Zarr shard. +**[Intent](index.qmd#intent).** Distribution and visualization. Both are served per scene, which is why this case does not need a cube. + +**The path.** Existing data, and the producer operates the processing chain, so this is a free choice. The topology opens `{COG, GeoZarr}` and the [intent](index.qmd#intent) decides between them. Per-scene **distribution and visualization** favors COG. **Time-series analysis** favors a GeoZarr cube. The [grouping lever](index.qmd#the-object-size-and-grouping-lever) is COG internal tiling, or a scene-size Zarr shard. **Measured, one scene.** Both layouts cleared the warm tier. COG came to 2625.7 MiB in 28 objects. A scene-size GeoZarr shard came to 1975.6 MiB in 26 objects, about 25 % smaller. That gap is the **codec**, Deflate against ZSTD, not the format. Against the 2023.7 MiB delivered `.zip`, the GeoZarr store is about source size (−2 %) and the COG is larger (+30 %), because the COG carries overviews. -A **too-small 2048² shard** split the same bytes into 843 objects that fell below the tier. This is exactly the object-grouping anti-pattern the framework warns about. +**[Interpretation encoding](index.qmd#interpretation-encoding) is a real differentiator here.** The source reflectance is stored as scaled integers, and the nodata fill is left undeclared. COG puts the scale, the offset, and the nodata in side metadata that readers honor unevenly. Zarr v3 can carry all three as first-class array properties, using the `scale_offset` codec and a declared fill value, so every reader recovers the same physical value. + +**Where it lands.** COG per scene for distribution and visualization. + +For temporal analysis GeoZarr is the right target, but **the arm measured here does not yet deliver it.** What was written is a scene-size store: one scene, so there is no time dimension to exploit. It proves the container and the encoding, because it comes out at roughly source size and it carries the scale, offset, and nodata correctly. A time-series query against it still has to open one store per date, which is the same fan-out the COG layout has. + +The temporal benefit appears only once many scenes are concatenated along time into a single cube. That is the regime measured in [case 2](#case-time-cube), and it is a collection-level layout decision, not something a scene-size store can give you. Read the per-scene store as a building block and the cube as the product. -**Interpretation encoding is a real differentiator here.** The source reflectance is stored as scaled integers, and the nodata fill is left undeclared. COG puts the scale, the offset, and the nodata in side metadata that readers honor unevenly. Zarr v3 can carry all three as first-class array properties, using the `scale_offset` codec and a declared fill value, so every reader recovers the same physical value. +The associated masking product, which is written as many small GeoTIFFs, is a clear anti-pattern to fix at the source. -**Where it lands.** COG per scene for distribution and visualization. A scene-size GeoZarr shard for temporal analysis. The associated masking product, which is written as many small GeoTIFFs, is a clear anti-pattern to fix at the source. +## 2. Raster time-cube: GeoZarr, with COG as the alternative {#case-time-cube} -## 2. Raster time-cube: Zarr v3, with COG as the alternative +**Product:** Sentinel-1 RTC. **Topology:** 2D raster over time. **Source format:** geoTiff in a `.zip`. **Candidate set:** `{COG, GeoZarr}`. -**Product:** Sentinel-1 RTC. **Topology:** 2D raster over time. **Candidate set:** `{COG, GeoZarr}`. +**[Intent](index.qmd#intent).** Advanced analysis and visualization lead, with distribution close behind but weighted a little lower. The analysis half is what separates this case from case 1: it is asked over time, not per scene, and that is what pulls the answer to a cube. **The path.** Existing data, reprocessable, so a free choice. A year of RTC over a region is naturally a `time × y × x × pol` cube. On a Zarr cube, a time-series slice or an area-of-interest subset is one range-addressable query. Across thousands of per-timestamp COGs, the same request becomes a large fan-out. Sharding reconciles fine-grained chunk access with the backend's preference for large objects. @@ -60,7 +68,9 @@ Both issues are being fixed for the final campaign, so that display is compared ## 3. Point cloud: netCDF to COPC -**Product:** SWOT L2_HR Pixel Cloud. **Topology:** point cloud. **Candidate set:** `{COPC, GeoParquet}`. +**Product:** SWOT L2_HR Pixel Cloud. **Topology:** point cloud. **Source format:** netCDF. **Candidate set:** `{COPC, GeoParquet}`. + +**[Intent](index.qmd#intent).** Distribution, visualization, and data fusion. The fusion intent is the one that matters most here: these points get combined with other water products, so a reader has to recover the same physical value per point and subset the same area cheaply. That is what makes the missing spatial index in the source a real cost rather than an inconvenience. **The path.** Existing data, reprocessable, so a free choice. The point-cloud topology gives COPC, which is a clustered octree inside a LAZ file. The netCDF source has no spatial index and no level of detail, so reading a small geographic subset means scanning the whole granule. @@ -79,7 +89,9 @@ An earlier measurement on this product looked like a 97 % size saving. It had dr ## 4. Vector and tabular: shapefile to GeoParquet -**Product:** SWOT L2_HR Lake Single Pass. **Topology:** vector and tabular. **Candidate set:** `{GeoParquet, FlatGeobuf}`. +**Product:** SWOT L2_HR Lake Single Pass. **Topology:** vector and tabular. **Source format:** shapefile. **Candidate set:** `{GeoParquet, FlatGeobuf}`. + +**[Intent](index.qmd#intent).** Distribution, download, and data fusion. Note that visualization is *absent* from this mix, and download is present. That combination is why the answer is not automatically the columnar option: download rewards a single streamable file, which is FlatGeobuf's strength, while fusion rewards cheap joins on a fixed schema, which is GeoParquet's. The fusion weight is what settles it. **The path.** Existing data, reprocessable, so a free choice. This is a fixed-schema tabular product, with a lake id, a reach id, water-surface elevation, area, and more. That is the native fit for GeoParquet, with its columns, row groups, and spatial partitioning. A sparse Zarr store would add complexity with no gain. @@ -91,7 +103,9 @@ Each pass sits below the warm-tier floor. The lever is therefore **aggregation a ## 5. Small N-D grid: the aggregation regime -**Product:** SWOT L2_HR Raster. **Topology:** N-D grid. **Candidate set:** `{GeoZarr, COG}`. +**Product:** SWOT L2_HR Raster. **Topology:** N-D grid. **Source format:** geoTiff. **Candidate set:** `{GeoZarr, COG}`. + +**[Intent](index.qmd#intent).** Distribution and download, with data fusion against the other SWOT products, and some visualization. This mix is provisional and still to be confirmed with the product owner. **The path.** Existing data, reprocessable, so a free choice. A 100 m raster granule covers a small tile, about 160 km across. @@ -119,7 +133,7 @@ This case is also where the ["unlock this"](index.qmd#dead-ends-are-unlock-this) | Case | Topology | Lands on | Size against source | Main gain | | --- | --- | --- | --- | --- | -| 1. Optical scene | 2D raster | COG, or GeoZarr for cubes | COG +30 %, GeoZarr −2 % | Partial access, encoding | +| 1. Optical scene | 2D raster | COG per scene; a cube needs cross-scene aggregation | COG +30 %, GeoZarr −2 % | Partial access, encoding | | 2. SAR time-cube | Raster over time | GeoZarr v3 sharded | COG +47 %, GeoZarr +10 % | One query instead of a fan-out | | 3. Pixel cloud | Point cloud | COPC | −16 % | Spatial index, level of detail | | 4. Lake vector | Vector, tabular | GeoParquet, aggregated | +14 % | Row-group bbox index | diff --git a/decision-framework/index.qmd b/decision-framework/index.qmd index 493f8303..11065b9a 100644 --- a/decision-framework/index.qmd +++ b/decision-framework/index.qmd @@ -96,12 +96,13 @@ Apply these within the candidate set your topology opened. ### Intent -What is the data for? Archive and download, processing, visualization, or distribution. Two points matter: +What is the data for? Archive and download, processing, visualization, distribution, or data fusion. Three points matter: - **Visualization is a first-class intent**, not a side effect. It has its own layout needs, mainly [overviews](../glossary.qmd#overviews) or a level of detail. - **On-the-fly and batched processing pull the layout in opposite directions.** On-the-fly access rewards small chunks, fine overviews, and cache locality. Batched processing rewards large shards or row groups and a layout that reads well in sequence. +- **Data fusion raises the stakes on encoding.** Combining products from different missions or processors means every input must be read back as the same physical value, and subsets from each must line up. Fusion is a form of processing, but it is worth naming separately: it weights [interpretation encoding](#interpretation-encoding) and consistent grids far higher than single-product analysis does, and it usually rewards a tabular or columnar target where joins are cheap. -Most real datasets serve several intents at once. Weigh them in proportion rather than picking one. +Most real datasets serve several intents at once. Weigh them in proportion rather than picking one. The [case studies](case-studies.qmd) state an intent mix for each product, which is what decides the ranking inside a candidate set. ### The object-size and grouping lever From 37a076d357cedad756ffb6c88fe45c76c8b9bb1b Mon Sep 17 00:00:00 2001 From: Emmanuel Mathot Date: Wed, 5 Aug 2026 08:13:42 +0200 Subject: [PATCH 07/59] Update decision-framework/index.qmd Co-authored-by: Pete Gadomski --- decision-framework/index.qmd | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/decision-framework/index.qmd b/decision-framework/index.qmd index 11065b9a..bdc9d4bd 100644 --- a/decision-framework/index.qmd +++ b/decision-framework/index.qmd @@ -33,7 +33,7 @@ The framework does not restate that definition. It assumes it. Two consequences **P1. The format must support partial and parallel reads over HTTP.** [Random access](../glossary.qmd#random-access) through [HTTP range requests](../glossary.qmd#http-range-request) is what makes a format usable from [object storage](../glossary.qmd#object-storage-cloud-storage). It depends on the rest of the pattern: [metadata](../glossary.qmd#metadata) in a consistent place that a reader can load in a single read, and then addressable [chunks](../glossary.qmd#chunk), internal tiling, or both. A reader that cannot locate the bytes it needs from the metadata alone cannot fetch only those bytes. -**P2. No opaque bundles.** An [archive format](../glossary.qmd#archive-format) is not disqualified just for being an archive. [PMTiles](../pmtiles/intro.qmd) is an archive and it is cloud-optimized, because its internal directory makes each tile addressable. What disqualifies a candidate is an archive with no addressable internal layout, or a layer of [external compression](../glossary.qmd#external-compression) wrapped around data that was already cloud-optimized. A zipped Zarr store, a monolithic LAZ file, and a single Parquet file with no row groups all fail this way: the client must fetch the whole object before it can use any part of it. +**P2. No opaque bundles.** An [archive format](../glossary.qmd#archive-format) is not disqualified just for being an archive. [PMTiles](../pmtiles/intro.qmd) is an archive and it is cloud-optimized, because its internal directory makes each tile addressable. What disqualifies a candidate is an archive with no addressable internal layout, or a layer of [external compression](../glossary.qmd#external-compression) wrapped around data that was already cloud-optimized. A zipped Zarr store, a LAZ file without [COPC metadata](https://copc.io/), and a single Parquet file with no row groups all fail this way: the client must fetch the whole object before it can use any part of it. P2 is easy to fail by accident, and it is worth checking separately from object size. A bucket can hold large, well-sized objects and still fail P2, because the format inside each object is opaque. Object size and internal structure are two independent axes, and a candidate must satisfy both. From ae6dbd5f6116c84c99a5e0da52391ff03dfbcd75 Mon Sep 17 00:00:00 2001 From: Emmanuel Mathot Date: Wed, 5 Aug 2026 08:14:38 +0200 Subject: [PATCH 08/59] Update decision-framework/index.qmd Co-authored-by: Pete Gadomski --- decision-framework/index.qmd | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/decision-framework/index.qmd b/decision-framework/index.qmd index bdc9d4bd..f6c5f7a8 100644 --- a/decision-framework/index.qmd +++ b/decision-framework/index.qmd @@ -41,7 +41,7 @@ P2 is easy to fail by accident, and it is worth checking separately from object Two questions come first. They apply to all geospatial data. -1. **Control.** Do you have existing data, and can you reprocess it? This decides whether you can choose freely, whether you must virtualize, or whether you have reached a dead end. +1. **Control.** Do the data already exist? If they exist, are you able (technically, legally) to reprocess them? 2. **Topology.** What shape is the data? This selects the **format family**. It opens a candidate set. It does not pick one format. Three refinements then rank the candidates: **intent**, the **object-size and grouping lever**, and **ecosystem maturity**. A fourth axis, **interpretation encoding**, is worth a separate look, because the formats differ a lot on it. From 69049838fb49589f6bee5f47f68e3c4650609310 Mon Sep 17 00:00:00 2001 From: Emmanuel Mathot Date: Wed, 5 Aug 2026 08:15:31 +0200 Subject: [PATCH 09/59] Update decision-framework/index.qmd Co-authored-by: Pete Gadomski --- decision-framework/index.qmd | 1 - 1 file changed, 1 deletion(-) diff --git a/decision-framework/index.qmd b/decision-framework/index.qmd index f6c5f7a8..f80f9eac 100644 --- a/decision-framework/index.qmd +++ b/decision-framework/index.qmd @@ -35,7 +35,6 @@ The framework does not restate that definition. It assumes it. Two consequences **P2. No opaque bundles.** An [archive format](../glossary.qmd#archive-format) is not disqualified just for being an archive. [PMTiles](../pmtiles/intro.qmd) is an archive and it is cloud-optimized, because its internal directory makes each tile addressable. What disqualifies a candidate is an archive with no addressable internal layout, or a layer of [external compression](../glossary.qmd#external-compression) wrapped around data that was already cloud-optimized. A zipped Zarr store, a LAZ file without [COPC metadata](https://copc.io/), and a single Parquet file with no row groups all fail this way: the client must fetch the whole object before it can use any part of it. -P2 is easy to fail by accident, and it is worth checking separately from object size. A bucket can hold large, well-sized objects and still fail P2, because the format inside each object is opaque. Object size and internal structure are two independent axes, and a candidate must satisfy both. ## The spine From 4413be723d08313002a8985c0255955472a31328 Mon Sep 17 00:00:00 2001 From: Emmanuel Mathot Date: Wed, 5 Aug 2026 08:34:19 +0200 Subject: [PATCH 10/59] Refine decision framework by clarifying the two-stage process for format selection and enhancing the explanation of topology and control stages. --- decision-framework/index.qmd | 29 ++++++++++++++++++++++++----- 1 file changed, 24 insertions(+), 5 deletions(-) diff --git a/decision-framework/index.qmd b/decision-framework/index.qmd index f80f9eac..0787d000 100644 --- a/decision-framework/index.qmd +++ b/decision-framework/index.qmd @@ -38,12 +38,31 @@ The framework does not restate that definition. It assumes it. Two consequences ## The spine -Two questions come first. They apply to all geospatial data. +The spine runs in **two stages**, and the topology question is the hinge between them. -1. **Control.** Do the data already exist? If they exist, are you able (technically, legally) to reprocess them? -2. **Topology.** What shape is the data? This selects the **format family**. It opens a candidate set. It does not pick one format. +**Stage 1 — control: do you have a choice?** Do the data already exist? If they +exist, are you able (technically, legally) to reprocess them? This stage can end +the process. If you cannot reprocess the data and cannot virtualize them, there +is no format decision to make yet. -Three refinements then rank the candidates: **intent**, the **object-size and grouping lever**, and **ecosystem maturity**. A fourth axis, **interpretation encoding**, is worth a separate look, because the formats differ a lot on it. +**Stage 2 — topology and refinement: which format?** You reach this stage only +when stage 1 leaves you free to choose. Here the shape of the data selects a +**format family**: the small group of cloud-optimized formats built for that +shape. A *family*, not a format. Two or three candidates normally come through, +and the refinements below rank them. + +The hand-off is worth stating outright, because the diagram can be read as if +the shape of the data were the answer: + +| Before the topology question | After it | +| --- | --- | +| Do I have a cloud-optimized target at all? | Which of these targets suits me best? | +| The outcome can be "not yet possible" | Every remaining candidate works; you are ranking | +| Decided by ownership and reprocessing rights | Decided by intent, layout, maturity, encoding | + +Three refinements then rank the candidates: **intent**, the **object-size and +grouping lever**, and **ecosystem maturity**. A fourth axis, **interpretation +encoding**, is worth a separate look, because the formats differ a lot on it. ```{mermaid} flowchart TD @@ -76,7 +95,7 @@ flowchart TD **Neither.** Some sources cannot be optimized or virtualized today. Examples are HDF4, encrypted files, and a single large uncompressed blob. See [Dead ends are "unlock this"](#dead-ends-are-unlock-this). -### The topology gate +### What shape are the data? The shape of the data opens a candidate set: From fc422dc73984e4501859984c8722efecfc00e83f Mon Sep 17 00:00:00 2001 From: Emmanuel Mathot Date: Wed, 5 Aug 2026 09:45:50 +0200 Subject: [PATCH 11/59] Enhance decision framework documentation by adding key terms and clarifying definitions for improved understanding and context. --- decision-framework/index.qmd | 34 +++++++++++++++++++++++++--------- 1 file changed, 25 insertions(+), 9 deletions(-) diff --git a/decision-framework/index.qmd b/decision-framework/index.qmd index 0787d000..636de79f 100644 --- a/decision-framework/index.qmd +++ b/decision-framework/index.qmd @@ -25,6 +25,22 @@ This is a first, deliberately simple version. It aligns on the questions to ask - It does not replace a benchmark. When two candidates are close, measure them. ::: +## Key terms + +The framework uses a few terms in a specific way. Each links to the section that develops it. + +| Term | What it means here | +| --- | --- | +| [**Archival-format mandate**](#control) | A requirement to keep a specific archival format for preservation, such as HDF5, netCDF or GRIB, whatever cloud-optimized format you publish alongside it. | +| [**Control**](#control) | Whether the choice is yours to make at all: do the data already exist, and may you reprocess them, technically and legally. This is stage 1 of the spine, and it can end in "not yet possible". | +| [**Ecosystem maturity**](#maturity) | How settled a format is, on an L1 to L5 scale from experimental to de-facto standard. It answers "safe default, or bet?". | +| [**Grouping lever**](#grouping-lever) | The knob each format gives you for grouping bytes into objects: COG internal tiling, Zarr v3 sharding, COPC octree nodes, GeoParquet row groups. | +| [**Intent**](#intent) | What the data are for: archive and download, processing, visualization, distribution, or data fusion. Most products serve several at once, in proportion. | +| [**Interpretation encoding**](#interpretation-encoding) | Whether the format carries units, CRS, nodata and any scale and offset transform in-format, so every reader recovers the same physical value, or leans on a side catalog. | +| [**Object size**](#grouping-lever) | The size of the objects your layout actually writes to storage. Object stores reward a larger average object, and many backends apply a size policy or storage tier. | +| [**Refinement**](#refinements) | A criterion applied *inside* a format family to rank its members. There are four: intent, the grouping lever, ecosystem maturity, and interpretation encoding. | +| [**Topology**](#topology) | The shape of the data: 2D raster, N-D cube, point cloud, or vector and tabular. Answering *what shape are the data?* selects a format family, not a format. | + ## Preconditions This guide already defines what makes a format cloud-optimized. The Welcome page states it as a four-part [cloud-optimized format pattern](../index.qmd#the-opportunity), and the glossary defines [cloud-optimized](../glossary.qmd#cloud-optimized) as the ability to read a meaningful *part* of a file without downloading *all* of it. @@ -69,9 +85,9 @@ flowchart TD A[Data to publish or migrate] --> B{Existing data?} B -- No --> C{Archival-format mandate?} C -- Yes --> C1[Cloud-optimized HDF5 guidance + pick a CN access format] - C -- No --> T[Topology gate] + C -- No --> T[What shape are the data?] B -- Yes --> D{Can you reprocess?} - D -- Yes --> T + D -- Yes --> T[What shape are the data?] D -- No --> E{Optimizable / virtualizable?} E -- Yes --> E1[VirtualiZarr / Kerchunk / Icechunk: limited, better than nothing] E -- No --> E2[Dead end: not yet funded, unlock this] @@ -85,7 +101,7 @@ flowchart TD T4 --> R ``` -### The control fork +### The control fork {#control} **No existing data.** You are designing a new product, so you can choose freely. Go to the topology gate. If a mandate requires you to keep an archival format such as HDF5 or netCDF, you have two jobs: apply the [cloud-optimized HDF/NetCDF](../cloud-optimized-netcdf4-hdf5/index.qmd) guidance to the archive, and choose a cloud-optimized access format for the data you publish. @@ -95,7 +111,7 @@ flowchart TD **Neither.** Some sources cannot be optimized or virtualized today. Examples are HDF4, encrypted files, and a single large uncompressed blob. See [Dead ends are "unlock this"](#dead-ends-are-unlock-this). -### What shape are the data? +### Topology: what shape are the data? {#topology} The shape of the data opens a candidate set: @@ -108,11 +124,11 @@ The shape of the data opens a candidate set: The gate is about topology, not about the format the data arrives in. A netCDF granule that holds a point cloud belongs on the point-cloud branch. See the [format table](../index.qmd#data-type-to-traditional-to-cloud-optimized-geospatial-file-format-table) for how traditional formats map to cloud-optimized ones. -## Refinements +## Refinements {#refinements} Apply these within the candidate set your topology opened. -### Intent +### Intent {#intent} What is the data for? Archive and download, processing, visualization, distribution, or data fusion. Three points matter: @@ -122,7 +138,7 @@ What is the data for? Archive and download, processing, visualization, distribut Most real datasets serve several intents at once. Weigh them in proportion rather than picking one. The [case studies](case-studies.qmd) state an intent mix for each product, which is what decides the ranking inside a candidate set. -### The object-size and grouping lever +### The object-size and grouping lever {#grouping-lever} Every candidate format has a knob that sets how bytes are grouped into objects: @@ -142,7 +158,7 @@ The lever works in both directions, and which direction you need depends on prod - **Products that are already large.** Group *within* one product. Size the shard, the tile, or the row group so it clears the policy. - **Products that are small.** Sharding within one granule only makes the fragmentation worse. Group *above* the product instead: mosaic or concatenate many granules into one range-addressable object. -### Ecosystem maturity +### Ecosystem maturity {#maturity} The real question is not a version number. It is whether you need something production-grade that you can count on, or you are willing to be an early adopter. Maturity is what separates a safe default from a bet. @@ -185,7 +201,7 @@ An L2 candidate can still be the right answer. The N-D cube branch is exactly th Read maturity against your own tolerance. A one-off internal product and a mission archive with a twenty-year retention commitment can reasonably land on different levels from the same candidate set. -### Interpretation encoding +### Interpretation encoding {#interpretation-encoding} Does the format carry the information a reader needs to recover the same physical value, or does it lean on a side catalog? The information is units, the [CRS](../glossary.qmd#coordinate-reference-system-crs), the nodata fill, and any scale and offset transform. From b9a385ec37d0200bde0e86a9e99b5f2a481c480f Mon Sep 17 00:00:00 2001 From: Emmanuel Mathot Date: Wed, 5 Aug 2026 10:00:00 +0200 Subject: [PATCH 12/59] Clarify terminology and references in decision framework documentation for improved accuracy and understanding. --- decision-framework/case-studies.qmd | 4 ++-- decision-framework/index.qmd | 16 +++++++++------- 2 files changed, 11 insertions(+), 9 deletions(-) diff --git a/decision-framework/case-studies.qmd b/decision-framework/case-studies.qmd index 67d15af5..a1835f48 100644 --- a/decision-framework/case-studies.qmd +++ b/decision-framework/case-studies.qmd @@ -7,7 +7,7 @@ Each case study below runs a real product through the [decision framework](index Read each case in two parts: -1. **The path through the spine**: the control fork, the topology gate, then intent, the object-size lever, and maturity. +1. **The path through the spine**: the control fork, the [topology question](index.qmd#topology), then intent, the object-size lever, and maturity. 2. **What the measurement showed.** The examples use public Copernicus and NASA missions. The [CNES Datalake](https://cnes.fr/en/projects/centre-de-calcul) appears as one worked example of a tiered object store. It is not the only kind of backend, and its tier numbers are not a general rule. @@ -34,7 +34,7 @@ They were produced with [`cng-formats-benchmark`](https://github.com/development **[Intent](index.qmd#intent).** Distribution and visualization. Both are served per scene, which is why this case does not need a cube. -**The path.** Existing data, and the producer operates the processing chain, so this is a free choice. The topology opens `{COG, GeoZarr}` and the [intent](index.qmd#intent) decides between them. Per-scene **distribution and visualization** favors COG. **Time-series analysis** favors a GeoZarr cube. The [grouping lever](index.qmd#the-object-size-and-grouping-lever) is COG internal tiling, or a scene-size Zarr shard. +**The path.** Existing data, and the producer operates the processing chain, so this is a free choice. The topology opens `{COG, GeoZarr}` and the [intent](index.qmd#intent) decides between them. Per-scene **distribution and visualization** favors COG. **Time-series analysis** favors a GeoZarr cube. The [grouping lever](index.qmd#grouping-lever) is COG internal tiling, or a scene-size Zarr shard. **Measured, one scene.** Both layouts cleared the warm tier. COG came to 2625.7 MiB in 28 objects. A scene-size GeoZarr shard came to 1975.6 MiB in 26 objects, about 25 % smaller. That gap is the **codec**, Deflate against ZSTD, not the format. Against the 2023.7 MiB delivered `.zip`, the GeoZarr store is about source size (−2 %) and the COG is larger (+30 %), because the COG carries overviews. diff --git a/decision-framework/index.qmd b/decision-framework/index.qmd index 636de79f..d497e29f 100644 --- a/decision-framework/index.qmd +++ b/decision-framework/index.qmd @@ -84,13 +84,13 @@ encoding**, is worth a separate look, because the formats differ a lot on it. flowchart TD A[Data to publish or migrate] --> B{Existing data?} B -- No --> C{Archival-format mandate?} - C -- Yes --> C1[Cloud-optimized HDF5 guidance + pick a CN access format] + C -- Yes --> C1[Optimize the mandated format + publish a cloud-optimized copy] C -- No --> T[What shape are the data?] B -- Yes --> D{Can you reprocess?} D -- Yes --> T[What shape are the data?] D -- No --> E{Optimizable / virtualizable?} E -- Yes --> E1[VirtualiZarr / Kerchunk / Icechunk: limited, better than nothing] - E -- No --> E2[Dead end: not yet funded, unlock this] + E -- No --> E2[Dead end: no Cloud-Native formats work for your use-case] T --> T1[2D raster, COG] T --> T2[N-D cube, Zarr v3 / GeoZarr] T --> T3[Point cloud, COPC] @@ -103,9 +103,11 @@ flowchart TD ### The control fork {#control} -**No existing data.** You are designing a new product, so you can choose freely. Go to the topology gate. If a mandate requires you to keep an archival format such as HDF5 or netCDF, you have two jobs: apply the [cloud-optimized HDF/NetCDF](../cloud-optimized-netcdf4-hdf5/index.qmd) guidance to the archive, and choose a cloud-optimized access format for the data you publish. +**No existing data.** You are designing a new product, so you can choose freely. Go to [topology](#topology). -**Existing data you can reprocess.** You control the production chain, so you can also choose freely. Go to the topology gate. +**An archival-format mandate adds a second job, it does not remove that freedom.** If policy requires you to preserve a particular format like HDF5, netCDF, GRIB, or something specific to your domain, you still publish a cloud-optimized copy for readers, and you additionally make the mandated format as cloud-friendly as it can be. Where the mandated format is HDF5 or netCDF, this guide has [specific guidance](../cloud-optimized-netcdf4-hdf5/index.qmd) on consolidated metadata and chunk sizing. For other archival formats the principle holds even where the guide has no page: optimize what you must keep, and publish what people will actually read. + +**Existing data you can reprocess.** You control the production chain, so you can also choose freely. Go to the [topology question](#topology). **Existing data you cannot reprocess.** You can often still publish a virtual view over the bytes you have. [Virtual Zarr](../zarr/virtual-zarr.qmd), with VirtualiZarr, Kerchunk, or Icechunk, builds an index that points into the original files. This gives range-based access without rewriting the data. The result is more limited than a native store, but it is much better than nothing. @@ -122,7 +124,7 @@ The shape of the data opens a candidate set: | Point cloud | [COPC](../copc/index.qmd) | [GeoParquet](../geoparquet/index.qmd), if columnar analytics dominate | | Vector or tabular | [GeoParquet](../geoparquet/index.qmd) | [FlatGeobuf](../flatgeobuf/intro.qmd) for streaming, [PMTiles](../pmtiles/intro.qmd) for tiled delivery | -The gate is about topology, not about the format the data arrives in. A netCDF granule that holds a point cloud belongs on the point-cloud branch. See the [format table](../index.qmd#data-type-to-traditional-to-cloud-optimized-geospatial-file-format-table) for how traditional formats map to cloud-optimized ones. +The question is about topology, not about the format the data arrives in. A netCDF granule that holds a point cloud belongs on the point-cloud branch. See the [format table](../index.qmd#data-type-to-traditional-to-cloud-optimized-geospatial-file-format-table) for how traditional formats map to cloud-optimized ones. ## Refinements {#refinements} @@ -190,7 +192,7 @@ Where the candidates sit today: This distinction is easy to miss, and it decides how much risk you actually take on. -- **Zarr v3, the substrate, is dependable.** The core specification was finalized in 2024. [Sharding](#the-object-size-and-grouping-lever), the feature that matters most for object storage, is stable in `zarr-python` 3.x and readable by GDAL and xarray. Judged on its own, the Zarr v3 core behaves like an L4. +- **Zarr v3, the substrate, is dependable.** The core specification was finalized in 2024. [Sharding](#grouping-lever), the feature that matters most for object storage, is stable in `zarr-python` 3.x and readable by GDAL and xarray. Judged on its own, the Zarr v3 core behaves like an L4. - **GeoZarr, the convention layer, is still being ratified.** It is what makes a generic [Zarr](../glossary.qmd#zarr) store *geospatial*: CRS, dimension semantics, and multiscale pyramids. It does for Zarr roughly what the GeoTIFF tags do for a plain TIFF. It is a draft in an OGC Standards Working Group, on the standards track but not yet an adopted OGC standard, and the published draft can lag behind active convention work. Check [geozarr.org](https://geozarr.org/) for the current status before you rely on a specific version. So the L2 above belongs to the convention layer, not to Zarr. The leading reference implementations today, such as ESA's EOPF Sentinel output, are demonstrators rather than production guarantees. @@ -218,7 +220,7 @@ These are not gates. They apply whatever you choose. ### Metadata and cataloging -Metadata that a reader can fetch in a single request is part of [P1](#preconditions), not an extra. Each candidate in the topology gate has its own mechanism for it: Zarr consolidated metadata, COG headers, GeoParquet footers, and the COPC header. What this section adds is the level *above* a single file. +Metadata that a reader can fetch in a single request is part of [P1](#preconditions), not an extra. Each candidate in the [topology formats](#topology) has its own mechanism for it: Zarr consolidated metadata, COG headers, GeoParquet footers, and the COPC header. What this section adds is the level *above* a single file. At a scale of tens of thousands to millions of files, collection-level discovery becomes critical. [STAC](https://stacspec.org/) has an essential role there, and it is part of the cloud-native geospatial ecosystem that we recommend. It is **not a format and not a gate in this framework**, so this page does not prescribe it. Choose a catalog approach that fits your organization. From c0b8e4142672454f9eceae4837f83a7ac4df6344 Mon Sep 17 00:00:00 2001 From: Emmanuel Mathot Date: Wed, 5 Aug 2026 10:15:09 +0200 Subject: [PATCH 13/59] Clarify decision framework by refining terminology and adding detailed explanations for reprocessing and virtualization processes. --- decision-framework/index.qmd | 33 ++++++++++++++++++++++++++++++++- 1 file changed, 32 insertions(+), 1 deletion(-) diff --git a/decision-framework/index.qmd b/decision-framework/index.qmd index d497e29f..5d6bd7ad 100644 --- a/decision-framework/index.qmd +++ b/decision-framework/index.qmd @@ -88,7 +88,7 @@ flowchart TD C -- No --> T[What shape are the data?] B -- Yes --> D{Can you reprocess?} D -- Yes --> T[What shape are the data?] - D -- No --> E{Optimizable / virtualizable?} + D -- No --> E{Virtualizable?} E -- Yes --> E1[VirtualiZarr / Kerchunk / Icechunk: limited, better than nothing] E -- No --> E2[Dead end: no Cloud-Native formats work for your use-case] T --> T1[2D raster, COG] @@ -113,6 +113,37 @@ flowchart TD **Neither.** Some sources cannot be optimized or virtualized today. Examples are HDF4, encrypted files, and a single large uncompressed blob. See [Dead ends are "unlock this"](#dead-ends-are-unlock-this). +#### How to tell which branch you are on + +"Can you reprocess?" and "can you virtualize?" have concrete tests, not judgement calls. + +**You can reprocess** if you have the processor, the inputs, and the budget to re-run +the production chain. Then format and layout are both free choices: go to +[topology](#topology). + +**You can repackage** if you cannot re-run the chain but you may rewrite the delivered +files. A plain TIFF becomes a COG, a LAS or LAZ file becomes a COPC, a shapefile or CSV +becomes GeoParquet. Repackaging is a byte-level rewrite rather than a science +reprocessing, so it is often permitted where reprocessing is not. Format is a free +choice; the content is fixed at whatever was delivered. + +**You can virtualize** if the bytes you have can be indexed as they are. +[Virtual Zarr](../zarr/virtual-zarr.qmd) publishes that index without copying the data. +Three tests, in order: + +1. **Is the data chunked internally?** This is the hard requirement. A virtual store + cannot change the source's chunk structure, so a single contiguous block virtualizes + into one enormous chunk and buys you no useful partial access. +2. **Are those chunks a workable size?** You inherit them exactly. Chunks that are far + too small or far too large stay that way. +3. **Is there a parser for the format?** Kerchunk and VirtualiZarr cover the common + archival formats, including netCDF4, HDF5, GRIB2, TIFF and FITS. + +**Otherwise you are at a dead end for now.** The usual blockers are HDF4, encrypted +files, a codec no parser can pass through, and a single uncompressed blob with no +internal chunking. See [Dead ends are "unlock this"](#dead-ends-are-unlock-this). + + ### Topology: what shape are the data? {#topology} The shape of the data opens a candidate set: From a579973d63259725bb71e9e6ae75a8f1181225c4 Mon Sep 17 00:00:00 2001 From: Emmanuel Mathot Date: Wed, 5 Aug 2026 10:31:36 +0200 Subject: [PATCH 14/59] Refine decision framework documentation by clarifying the interpretation encoding and enhancing the explanation of the two-stage process for format selection. Update theme configuration in Quarto settings for improved clarity. --- _quarto.yml | 2 +- decision-framework/case-studies.qmd | 2 +- decision-framework/index.qmd | 71 ++++++++++++++++++----------- 3 files changed, 47 insertions(+), 28 deletions(-) diff --git a/_quarto.yml b/_quarto.yml index 3200d1f1..fa7a485d 100644 --- a/_quarto.yml +++ b/_quarto.yml @@ -98,7 +98,7 @@ format: html: include-in-header: plausible.html theme: - light: [cosmo, theme.scss] # from https://github.com/sta210-s22/website/blob/main/_quarto.yml + light: cosmo # from https://github.com/sta210-s22/website/blob/main/_quarto.yml dark: [cosmo, theme-dark.scss] code-copy: true code-overflow: wrap diff --git a/decision-framework/case-studies.qmd b/decision-framework/case-studies.qmd index a1835f48..8c0f5985 100644 --- a/decision-framework/case-studies.qmd +++ b/decision-framework/case-studies.qmd @@ -7,7 +7,7 @@ Each case study below runs a real product through the [decision framework](index Read each case in two parts: -1. **The path through the spine**: the control fork, the [topology question](index.qmd#topology), then intent, the object-size lever, and maturity. +1. **The path through the spine**: the control fork, the [topology question](index.qmd#topology), then intent, the object-size lever, maturity, and interpretation encoding. 2. **What the measurement showed.** The examples use public Copernicus and NASA missions. The [CNES Datalake](https://cnes.fr/en/projects/centre-de-calcul) appears as one worked example of a tiered object store. It is not the only kind of backend, and its tier numbers are not a general rule. diff --git a/decision-framework/index.qmd b/decision-framework/index.qmd index 5d6bd7ad..84bef78e 100644 --- a/decision-framework/index.qmd +++ b/decision-framework/index.qmd @@ -51,7 +51,6 @@ The framework does not restate that definition. It assumes it. Two consequences **P2. No opaque bundles.** An [archive format](../glossary.qmd#archive-format) is not disqualified just for being an archive. [PMTiles](../pmtiles/intro.qmd) is an archive and it is cloud-optimized, because its internal directory makes each tile addressable. What disqualifies a candidate is an archive with no addressable internal layout, or a layer of [external compression](../glossary.qmd#external-compression) wrapped around data that was already cloud-optimized. A zipped Zarr store, a LAZ file without [COPC metadata](https://copc.io/), and a single Parquet file with no row groups all fail this way: the client must fetch the whole object before it can use any part of it. - ## The spine The spine runs in **two stages**, and the topology question is the hinge between them. @@ -76,42 +75,62 @@ the shape of the data were the answer: | The outcome can be "not yet possible" | Every remaining candidate works; you are ranking | | Decided by ownership and reprocessing rights | Decided by intent, layout, maturity, encoding | -Three refinements then rank the candidates: **intent**, the **object-size and -grouping lever**, and **ecosystem maturity**. A fourth axis, **interpretation -encoding**, is worth a separate look, because the formats differ a lot on it. +Four refinements then rank the candidates: **intent**, the **object-size and +grouping lever**, **ecosystem maturity**, and **interpretation encoding**. The +last one deserves a close look, because the formats differ on it more than +people expect. ```{mermaid} flowchart TD - A[Data to publish or migrate] --> B{Existing data?} - B -- No --> C{Archival-format mandate?} - C -- Yes --> C1[Optimize the mandated format + publish a cloud-optimized copy] - C -- No --> T[What shape are the data?] - B -- Yes --> D{Can you reprocess?} - D -- Yes --> T[What shape are the data?] + A([Data to publish or migrate]) --> B{Existing data?} + + B -- No --> C{Archival-format
mandate?} + C -- Yes --> C1[Optimize the mandated format
+ publish a cloud-optimized copy] + C -- No --> T{{What shape are the data?}} + + B -- Yes --> D{Can you
reprocess?} + D -- Yes --> T D -- No --> E{Virtualizable?} - E -- Yes --> E1[VirtualiZarr / Kerchunk / Icechunk: limited, better than nothing] - E -- No --> E2[Dead end: no Cloud-Native formats work for your use-case] - T --> T1[2D raster, COG] - T --> T2[N-D cube, Zarr v3 / GeoZarr] - T --> T3[Point cloud, COPC] - T --> T4[Vector / tabular, GeoParquet] - T1 --> R[Refine: intent, object-size lever, maturity, interpretation] + E -- Yes --> E1[VirtualiZarr / Kerchunk / Icechunk
limited, better than nothing] + E -- No --> E2[Not yet implemented
unlock this] + + T --> T1(2D raster
COG) + T --> T2(N-D cube
Zarr v3 / GeoZarr) + T --> T3(Point cloud
COPC) + T --> T4(Vector / tabular
GeoParquet) + + T1 --> R([Refine: intent, object-size lever,
maturity, interpretation]) T2 --> R T3 --> R T4 --> R + + classDef entry fill:#e7eefb,stroke:#3f6db3,stroke-width:2px,color:#152744 + classDef ask fill:#fff6dc,stroke:#ad7c00,stroke-width:2px,color:#4a3600 + classDef hinge fill:#dfeffa,stroke:#2a7fb8,stroke-width:3px,color:#0e3247 + classDef family fill:#e4f4ea,stroke:#39875f,stroke-width:2px,color:#12301e + classDef partial fill:#fdeadb,stroke:#c06520,stroke-width:2px,color:#48260a + classDef refine fill:#ece6f7,stroke:#7355a6,stroke-width:2px,color:#291d44 + + class A entry + class B,C,D,E ask + class T hinge + class T1,T2,T3,T4 family + class C1,E1,E2 partial + class R refine + ``` ### The control fork {#control} -**No existing data.** You are designing a new product, so you can choose freely. Go to [topology](#topology). +**No existing data.** You are designing a new product, so you can choose freely. Go to the [topology question](#topology). -**An archival-format mandate adds a second job, it does not remove that freedom.** If policy requires you to preserve a particular format like HDF5, netCDF, GRIB, or something specific to your domain, you still publish a cloud-optimized copy for readers, and you additionally make the mandated format as cloud-friendly as it can be. Where the mandated format is HDF5 or netCDF, this guide has [specific guidance](../cloud-optimized-netcdf4-hdf5/index.qmd) on consolidated metadata and chunk sizing. For other archival formats the principle holds even where the guide has no page: optimize what you must keep, and publish what people will actually read. +**If an archival-format mandate applies,** it adds a second job, it does not remove that freedom. If policy requires you to preserve a particular format like HDF5, netCDF, GRIB, or something specific to your domain, you still publish a cloud-optimized copy for readers, and you additionally make the mandated format as cloud-friendly as it can be. Where the mandated format is HDF5 or netCDF, this guide has [specific guidance](../cloud-optimized-netcdf4-hdf5/index.qmd) on consolidated metadata and chunk sizing. For other archival formats the principle holds even where the guide has no page: optimize what you must keep, and publish what people will actually read. **Existing data you can reprocess.** You control the production chain, so you can also choose freely. Go to the [topology question](#topology). **Existing data you cannot reprocess.** You can often still publish a virtual view over the bytes you have. [Virtual Zarr](../zarr/virtual-zarr.qmd), with VirtualiZarr, Kerchunk, or Icechunk, builds an index that points into the original files. This gives range-based access without rewriting the data. The result is more limited than a native store, but it is much better than nothing. -**Neither.** Some sources cannot be optimized or virtualized today. Examples are HDF4, encrypted files, and a single large uncompressed blob. See [Dead ends are "unlock this"](#dead-ends-are-unlock-this). +**Neither.** Some sources cannot be repackaged or virtualized today. Examples are HDF4, encrypted files, and a single large uncompressed blob. See [Dead ends are "unlock this"](#dead-ends-are-unlock-this). #### How to tell which branch you are on @@ -163,7 +182,7 @@ Apply these within the candidate set your topology opened. ### Intent {#intent} -What is the data for? Archive and download, processing, visualization, distribution, or data fusion. Three points matter: +What are the data for? Archive and download, processing, visualization, distribution, or data fusion. Three points matter: - **Visualization is a first-class intent**, not a side effect. It has its own layout needs, mainly [overviews](../glossary.qmd#overviews) or a level of detail. - **On-the-fly and batched processing pull the layout in opposite directions.** On-the-fly access rewards small chunks, fine overviews, and cache locality. Batched processing rewards large shards or row groups and a layout that reads well in sequence. @@ -247,13 +266,13 @@ If your product stores scaled integers, or if an undeclared fill value would cha ## Cross-cutting considerations -These are not gates. They apply whatever you choose. +These are not decision points. They apply whatever you choose. ### Metadata and cataloging -Metadata that a reader can fetch in a single request is part of [P1](#preconditions), not an extra. Each candidate in the [topology formats](#topology) has its own mechanism for it: Zarr consolidated metadata, COG headers, GeoParquet footers, and the COPC header. What this section adds is the level *above* a single file. +Metadata that a reader can fetch in a single request is part of [P1](#preconditions), not an extra. Every candidate the [topology question](#topology) opens has its own mechanism for it: Zarr consolidated metadata, COG headers, GeoParquet footers, and the COPC header. What this section adds is the level *above* a single file. -At a scale of tens of thousands to millions of files, collection-level discovery becomes critical. [STAC](https://stacspec.org/) has an essential role there, and it is part of the cloud-native geospatial ecosystem that we recommend. It is **not a format and not a gate in this framework**, so this page does not prescribe it. Choose a catalog approach that fits your organization. +At a scale of tens of thousands to millions of files, collection-level discovery becomes critical. [STAC](https://stacspec.org/) has an essential role there, and it is part of the cloud-native geospatial ecosystem that we recommend. It is **not a format and not a decision point in this framework**, so this page does not prescribe it. Choose a catalog approach that fits your organization. ### Within-format depth @@ -267,7 +286,7 @@ These are mature and in scope. The per-format pages cover them. More experimenta ### Dead ends are "unlock this" -When a source cannot be optimized or virtualized today, the honest answer is *"not yet implemented"*, not *"impossible"*. The blocker is usually missing tooling or missing funding, and both can change. +When a source cannot be repackaged or virtualized today, the honest answer is *"not yet implemented"*, not *"impossible"*. The blocker is usually a missing parser, an unsupported codec, or source files with no internal chunking at all. Every one of those can change. Frame the branch that way, and route the reader back to the options: reprocess if you can get the chain changed, virtualize if a reader exists, or convert on ingest. Say what is missing, so somebody can unlock it. @@ -281,7 +300,7 @@ The idea is a **weighted mean of criteria**, each expressed as a number from 0 t | --- | --- | | Qualitative, no benchmark needed | **use-case fit**: your intent mix, applied to a per-format suitability matrix | | | **maturity**: the L1 to L5 level above, mapped to a utility (`level / 5`) | -| | **interpretation encoding**: the encoding level above, mapped to a utility | +| | **interpretation encoding**: how much the format carries in-format, mapped to a utility | | Measured, needs a benchmark run | **size**: output volume against the source | | | **performance**: measured read, write, and display, blended by the intent mix | From f908d81e797af5fc04c1f7c1c93d75eea43a3f4e Mon Sep 17 00:00:00 2001 From: Emmanuel Mathot Date: Thu, 6 Aug 2026 16:36:01 +0200 Subject: [PATCH 15/59] Enhance decision framework by expanding format family options for data shapes and clarifying candidate selections for improved guidance on cloud-optimized formats. --- decision-framework/index.qmd | 21 +++++++++++---------- 1 file changed, 11 insertions(+), 10 deletions(-) diff --git a/decision-framework/index.qmd b/decision-framework/index.qmd index 84bef78e..d94be776 100644 --- a/decision-framework/index.qmd +++ b/decision-framework/index.qmd @@ -94,10 +94,10 @@ flowchart TD E -- Yes --> E1[VirtualiZarr / Kerchunk / Icechunk
limited, better than nothing] E -- No --> E2[Not yet implemented
unlock this] - T --> T1(2D raster
COG) - T --> T2(N-D cube
Zarr v3 / GeoZarr) - T --> T3(Point cloud
COPC) - T --> T4(Vector / tabular
GeoParquet) + T --> T1(2D raster
COG, Zarr, PMTiles) + T --> T2(N-D cube
GeoZarr, COG) + T --> T3(Point cloud
COPC, GeoParquet) + T --> T4(Vector / tabular
GeoParquet, FlatGeobuf, PMTiles) T1 --> R([Refine: intent, object-size lever,
maturity, interpretation]) T2 --> R @@ -165,14 +165,15 @@ internal chunking. See [Dead ends are "unlock this"](#dead-ends-are-unlock-this) ### Topology: what shape are the data? {#topology} -The shape of the data opens a candidate set: +The shape of the data selects a **format family**: the candidate set you refine within. It does not select a format. [Intent](#intent) is what decides between the members. -| Topology | Primary candidate | Common alternatives | +| Data shape | Format family | Where each member wins | | --- | --- | --- | -| 2D raster, per scene or per tile | [COG](../cloud-optimized-geotiffs/intro.qmd) | Zarr, if the product is really a cube | -| N-D cube, for example `time × y × x × band` | [Zarr v3 / GeoZarr](../zarr/intro.qmd) | COG per time step, if slices are read alone | -| Point cloud | [COPC](../copc/index.qmd) | [GeoParquet](../geoparquet/index.qmd), if columnar analytics dominate | -| Vector or tabular | [GeoParquet](../geoparquet/index.qmd) | [FlatGeobuf](../flatgeobuf/intro.qmd) for streaming, [PMTiles](../pmtiles/intro.qmd) for tiled delivery | +| 2D raster, per scene or per tile | [COG](../cloud-optimized-geotiffs/intro.qmd), [Zarr](../zarr/intro.qmd), [PMTiles](../pmtiles/intro.qmd) | COG for distribution and visualization; Zarr if the product is really a cube; PMTiles for pre-rendered tile delivery | +| N-D cube, for example `time × y × x × band` | [GeoZarr](../zarr/intro.qmd), [COG](../cloud-optimized-geotiffs/intro.qmd) | GeoZarr for analysis across the cube; COG per time step if slices are read alone | +| Point cloud | [COPC](../copc/index.qmd), [GeoParquet](../geoparquet/index.qmd) | COPC for spatial subsetting and level of detail; GeoParquet if columnar analytics dominate | +| Vector or tabular | [GeoParquet](../geoparquet/index.qmd), [FlatGeobuf](../flatgeobuf/intro.qmd), [PMTiles](../pmtiles/intro.qmd) | GeoParquet for joins and analytics; FlatGeobuf for streaming download; PMTiles for tiled web delivery | + The question is about topology, not about the format the data arrives in. A netCDF granule that holds a point cloud belongs on the point-cloud branch. See the [format table](../index.qmd#data-type-to-traditional-to-cloud-optimized-geospatial-file-format-table) for how traditional formats map to cloud-optimized ones. From 559c35f46f59af7015d3923a47db6ba6e5c73e14 Mon Sep 17 00:00:00 2001 From: Emmanuel Mathot Date: Thu, 6 Aug 2026 16:44:13 +0200 Subject: [PATCH 16/59] Clarify decision framework by distinguishing between size and shape of data objects, enhancing the explanation of access patterns and their impact on format performance. --- decision-framework/index.qmd | 17 +++++++++++++++-- 1 file changed, 15 insertions(+), 2 deletions(-) diff --git a/decision-framework/index.qmd b/decision-framework/index.qmd index d94be776..15b63d50 100644 --- a/decision-framework/index.qmd +++ b/decision-framework/index.qmd @@ -31,6 +31,7 @@ The framework uses a few terms in a specific way. Each links to the section that | Term | What it means here | | --- | --- | +| [**Access pattern**](#grouping-lever) | How readers will actually hit the data: a time series at one point, a spatial window at one date, a full scan. Intent implies an access pattern, and your layout either serves it or fights it. | | [**Archival-format mandate**](#control) | A requirement to keep a specific archival format for preservation, such as HDF5, netCDF or GRIB, whatever cloud-optimized format you publish alongside it. | | [**Control**](#control) | Whether the choice is yours to make at all: do the data already exist, and may you reprocess them, technically and legally. This is stage 1 of the spine, and it can end in "not yet possible". | | [**Ecosystem maturity**](#maturity) | How settled a format is, on an L1 to L5 scale from experimental to de-facto standard. It answers "safe default, or bet?". | @@ -153,8 +154,7 @@ Three tests, in order: 1. **Is the data chunked internally?** This is the hard requirement. A virtual store cannot change the source's chunk structure, so a single contiguous block virtualizes into one enormous chunk and buys you no useful partial access. -2. **Are those chunks a workable size?** You inherit them exactly. Chunks that are far - too small or far too large stay that way. +2. **Are those chunks a workable size?** You inherit both the size and the shape exactly, so the source's chunking decides which access patterns stay cheap. A store chunked for one access pattern cannot be virtualized into a store that is good for a different one. 3. **Is there a parser for the format?** Kerchunk and VirtualiZarr cover the common archival formats, including netCDF4, HDF5, GRIB2, TIFF and FITS. @@ -211,6 +211,19 @@ The lever works in both directions, and which direction you need depends on prod - **Products that are already large.** Group *within* one product. Size the shard, the tile, or the row group so it clears the policy. - **Products that are small.** Sharding within one granule only makes the fragmentation worse. Group *above* the product instead: mosaic or concatenate many granules into one range-addressable object. +**Size and shape are two different questions.** How *big* the objects are is a +storage-backend question, answered above. How the bytes are arranged *inside* them +(chunk shape, and the order of the dimensions) is an **access-pattern** question, +and the backend has no opinion on it. A `time × y × x` cube chunked long in time +serves a time series at one point cheaply and a full-scene map expensively; chunked +wide in `y` and `x` it does the opposite. This guide works it through, with a +diagram, in [chunk shape vs chunk size](../cloud-optimized-netcdf4-hdf5/index.qmd#chunk-shape-vs-chunk-size). + +Shape rarely decides *which* format you pick, because every member of a family +exposes some version of the knob. It decides how well the format you picked +performs. And it is mostly settled back at the [control](#control) question, because only reprocessing lets you choose the shape freely. + + ### Ecosystem maturity {#maturity} The real question is not a version number. It is whether you need something production-grade that you can count on, or you are willing to be an early adopter. Maturity is what separates a safe default from a bet. From a9ae111eeff4072dd3b928d89d638442f483a555 Mon Sep 17 00:00:00 2001 From: Emmanuel Mathot Date: Thu, 6 Aug 2026 16:45:29 +0200 Subject: [PATCH 17/59] Clarify terminology in decision framework by correcting pluralization of "N-D cube" to "N-D cubes" and enhancing consistency in data shape descriptions. --- decision-framework/index.qmd | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/decision-framework/index.qmd b/decision-framework/index.qmd index 15b63d50..056863d1 100644 --- a/decision-framework/index.qmd +++ b/decision-framework/index.qmd @@ -40,7 +40,7 @@ The framework uses a few terms in a specific way. Each links to the section that | [**Interpretation encoding**](#interpretation-encoding) | Whether the format carries units, CRS, nodata and any scale and offset transform in-format, so every reader recovers the same physical value, or leans on a side catalog. | | [**Object size**](#grouping-lever) | The size of the objects your layout actually writes to storage. Object stores reward a larger average object, and many backends apply a size policy or storage tier. | | [**Refinement**](#refinements) | A criterion applied *inside* a format family to rank its members. There are four: intent, the grouping lever, ecosystem maturity, and interpretation encoding. | -| [**Topology**](#topology) | The shape of the data: 2D raster, N-D cube, point cloud, or vector and tabular. Answering *what shape are the data?* selects a format family, not a format. | +| [**Topology**](#topology) | The shape of the data: 2D raster, N-D cubes, point cloud, or vector and tabular. Answering *what shape are the data?* selects a format family, not a format. | ## Preconditions @@ -96,7 +96,7 @@ flowchart TD E -- No --> E2[Not yet implemented
unlock this] T --> T1(2D raster
COG, Zarr, PMTiles) - T --> T2(N-D cube
GeoZarr, COG) + T --> T2(N-D cubes
GeoZarr, COG) T --> T3(Point cloud
COPC, GeoParquet) T --> T4(Vector / tabular
GeoParquet, FlatGeobuf, PMTiles) @@ -170,7 +170,7 @@ The shape of the data selects a **format family**: the candidate set you refine | Data shape | Format family | Where each member wins | | --- | --- | --- | | 2D raster, per scene or per tile | [COG](../cloud-optimized-geotiffs/intro.qmd), [Zarr](../zarr/intro.qmd), [PMTiles](../pmtiles/intro.qmd) | COG for distribution and visualization; Zarr if the product is really a cube; PMTiles for pre-rendered tile delivery | -| N-D cube, for example `time × y × x × band` | [GeoZarr](../zarr/intro.qmd), [COG](../cloud-optimized-geotiffs/intro.qmd) | GeoZarr for analysis across the cube; COG per time step if slices are read alone | +| N-D cubes, for example `time × y × x × band` | [GeoZarr](../zarr/intro.qmd), [COG](../cloud-optimized-geotiffs/intro.qmd) | GeoZarr for analysis across the cube; COG per time step if slices are read alone | | Point cloud | [COPC](../copc/index.qmd), [GeoParquet](../geoparquet/index.qmd) | COPC for spatial subsetting and level of detail; GeoParquet if columnar analytics dominate | | Vector or tabular | [GeoParquet](../geoparquet/index.qmd), [FlatGeobuf](../flatgeobuf/intro.qmd), [PMTiles](../pmtiles/intro.qmd) | GeoParquet for joins and analytics; FlatGeobuf for streaming download; PMTiles for tiled web delivery | @@ -263,7 +263,7 @@ So the L2 above belongs to the convention layer, not to Zarr. The leading refere #### Maturity is a warning, not a veto -An L2 candidate can still be the right answer. The N-D cube branch is exactly that case: there is no mature alternative with the same topology, so a low level does not mean "choose something else". It means **plan for change**. Budget for convention churn, pin the versions you depend on, and expect the visualization story in particular to improve as multiscale tooling catches up. +An L2 candidate can still be the right answer. The N-D cubes branch is exactly that case: there is no mature alternative with the same topology, so a low level does not mean "choose something else". It means **plan for change**. Budget for convention churn, pin the versions you depend on, and expect the visualization story in particular to improve as multiscale tooling catches up. Read maturity against your own tolerance. A one-off internal product and a mission archive with a twenty-year retention commitment can reasonably land on different levels from the same candidate set. From 2edda05527a1c62f186cdb2d2d853a4d867e0d0e Mon Sep 17 00:00:00 2001 From: Emmanuel Mathot Date: Mon, 10 Aug 2026 09:18:03 +0200 Subject: [PATCH 18/59] Clarify decision framework terminology by updating "unlock this" to "not yet possible" for improved accuracy in describing virtualization limitations. --- decision-framework/case-studies.qmd | 2 +- decision-framework/index.qmd | 10 +++++----- 2 files changed, 6 insertions(+), 6 deletions(-) diff --git a/decision-framework/case-studies.qmd b/decision-framework/case-studies.qmd index 8c0f5985..93a2d8ce 100644 --- a/decision-framework/case-studies.qmd +++ b/decision-framework/case-studies.qmd @@ -127,7 +127,7 @@ An immutable archive, for example HDF5 or netCDF that you cannot reprocess, can The result is more limited than a native store. Chunk sizes are whatever the source chose, you cannot add overviews, and you inherit the source's codec. It is still much better than whole-file download, and it is often the only option available. -This case is also where the ["unlock this"](index.qmd#dead-ends-are-unlock-this) framing matters. If the source cannot even be virtualized, name what is missing rather than declaring the case impossible. +This case is also where the ["not yet possible"](index.qmd#not-yet-possible) framing matters. If the source cannot even be virtualized, name what is missing rather than declaring the case impossible. ## Summary diff --git a/decision-framework/index.qmd b/decision-framework/index.qmd index 056863d1..1ce2f10e 100644 --- a/decision-framework/index.qmd +++ b/decision-framework/index.qmd @@ -93,7 +93,7 @@ flowchart TD D -- Yes --> T D -- No --> E{Virtualizable?} E -- Yes --> E1[VirtualiZarr / Kerchunk / Icechunk
limited, better than nothing] - E -- No --> E2[Not yet implemented
unlock this] + E -- No --> E2[Not yet possible
say what is missing] T --> T1(2D raster
COG, Zarr, PMTiles) T --> T2(N-D cubes
GeoZarr, COG) @@ -131,7 +131,7 @@ flowchart TD **Existing data you cannot reprocess.** You can often still publish a virtual view over the bytes you have. [Virtual Zarr](../zarr/virtual-zarr.qmd), with VirtualiZarr, Kerchunk, or Icechunk, builds an index that points into the original files. This gives range-based access without rewriting the data. The result is more limited than a native store, but it is much better than nothing. -**Neither.** Some sources cannot be repackaged or virtualized today. Examples are HDF4, encrypted files, and a single large uncompressed blob. See [Dead ends are "unlock this"](#dead-ends-are-unlock-this). +**Neither.** Some sources cannot be repackaged or virtualized today. Examples are HDF4, encrypted files, and a single large uncompressed blob. See [Not yet possible](#not-yet-possible). #### How to tell which branch you are on @@ -158,9 +158,9 @@ Three tests, in order: 3. **Is there a parser for the format?** Kerchunk and VirtualiZarr cover the common archival formats, including netCDF4, HDF5, GRIB2, TIFF and FITS. -**Otherwise you are at a dead end for now.** The usual blockers are HDF4, encrypted +**Otherwise there is no path yet.** The usual blockers are HDF4, encrypted files, a codec no parser can pass through, and a single uncompressed blob with no -internal chunking. See [Dead ends are "unlock this"](#dead-ends-are-unlock-this). +internal chunking. See [Not yet possible](#not-yet-possible). ### Topology: what shape are the data? {#topology} @@ -298,7 +298,7 @@ Some choices sit below the format and are foundational: These are mature and in scope. The per-format pages cover them. More experimental optimizations, such as bit-rounding and advanced compression tricks, are out of scope here, or should be marked as immature when you use them. -### Dead ends are "unlock this" +### Not yet possible {#not-yet-possible} When a source cannot be repackaged or virtualized today, the honest answer is *"not yet implemented"*, not *"impossible"*. The blocker is usually a missing parser, an unsupported codec, or source files with no internal chunking at all. Every one of those can change. From 5a3443c2965bffb94b460c63cbb9c948dfe42064 Mon Sep 17 00:00:00 2001 From: Emmanuel Mathot Date: Mon, 10 Aug 2026 09:22:52 +0200 Subject: [PATCH 19/59] Clarify virtualization limitations in decision framework by updating terminology from "impossible" to "not yet implemented" and providing context on HDF5's evolution to enhance reader understanding of format capabilities. --- decision-framework/index.qmd | 2 ++ 1 file changed, 2 insertions(+) diff --git a/decision-framework/index.qmd b/decision-framework/index.qmd index 1ce2f10e..2da8cb86 100644 --- a/decision-framework/index.qmd +++ b/decision-framework/index.qmd @@ -302,6 +302,8 @@ These are mature and in scope. The per-format pages cover them. More experimenta When a source cannot be repackaged or virtualized today, the honest answer is *"not yet implemented"*, not *"impossible"*. The blocker is usually a missing parser, an unsupported codec, or source files with no internal chunking at all. Every one of those can change. +The precedent is HDF5 itself: once the standard example of a format that could not be read efficiently from object storage, until a change in write strategy and reader tooling made it viable, see [Cloud-Optimized HDF/NetCDF](../cloud-optimized-netcdf4-hdf5/index.qmd). Formats move. + Frame the branch that way, and route the reader back to the options: reprocess if you can get the chain changed, virtualize if a reader exists, or convert on ingest. Say what is missing, so somebody can unlock it. ## Optional: make it quantitative From dda3e360dc23d454883f5dc2039c8f0b545f330b Mon Sep 17 00:00:00 2001 From: Emmanuel Mathot Date: Mon, 10 Aug 2026 09:27:26 +0200 Subject: [PATCH 20/59] Enhance decision framework by adding a repackage option for existing data and clarifying format family choices for N-D cubes, improving guidance on cloud-optimized formats. --- decision-framework/index.qmd | 15 ++++++++++----- 1 file changed, 10 insertions(+), 5 deletions(-) diff --git a/decision-framework/index.qmd b/decision-framework/index.qmd index 2da8cb86..5ced9764 100644 --- a/decision-framework/index.qmd +++ b/decision-framework/index.qmd @@ -91,12 +91,15 @@ flowchart TD B -- Yes --> D{Can you
reprocess?} D -- Yes --> T - D -- No --> E{Virtualizable?} + D -- No --> P{Can you repackage
the delivered files?} + P -- Yes --> P1[Format free,
content fixed as delivered] + P1 --> T + P -- No --> E{Virtualizable?} E -- Yes --> E1[VirtualiZarr / Kerchunk / Icechunk
limited, better than nothing] E -- No --> E2[Not yet possible
say what is missing] T --> T1(2D raster
COG, Zarr, PMTiles) - T --> T2(N-D cubes
GeoZarr, COG) + T --> T2(N-D cubes
GeoZarr, COG, CO-HDF5) T --> T3(Point cloud
COPC, GeoParquet) T --> T4(Vector / tabular
GeoParquet, FlatGeobuf, PMTiles) @@ -113,10 +116,10 @@ flowchart TD classDef refine fill:#ece6f7,stroke:#7355a6,stroke-width:2px,color:#291d44 class A entry - class B,C,D,E ask + class B,C,D,E,P ask class T hinge class T1,T2,T3,T4 family - class C1,E1,E2 partial + class C1,P1,E1,E2 partial class R refine ``` @@ -129,6 +132,8 @@ flowchart TD **Existing data you can reprocess.** You control the production chain, so you can also choose freely. Go to the [topology question](#topology). +**Existing data you cannot reprocess, but can rewrite.** You may still be allowed to repackage the delivered files: a plain TIFF becomes a COG, a LAZ file becomes a COPC, a shapefile becomes GeoParquet. The format is a free choice, but the content is fixed at whatever was delivered. Go to the [topology question](#topology). + **Existing data you cannot reprocess.** You can often still publish a virtual view over the bytes you have. [Virtual Zarr](../zarr/virtual-zarr.qmd), with VirtualiZarr, Kerchunk, or Icechunk, builds an index that points into the original files. This gives range-based access without rewriting the data. The result is more limited than a native store, but it is much better than nothing. **Neither.** Some sources cannot be repackaged or virtualized today. Examples are HDF4, encrypted files, and a single large uncompressed blob. See [Not yet possible](#not-yet-possible). @@ -170,7 +175,7 @@ The shape of the data selects a **format family**: the candidate set you refine | Data shape | Format family | Where each member wins | | --- | --- | --- | | 2D raster, per scene or per tile | [COG](../cloud-optimized-geotiffs/intro.qmd), [Zarr](../zarr/intro.qmd), [PMTiles](../pmtiles/intro.qmd) | COG for distribution and visualization; Zarr if the product is really a cube; PMTiles for pre-rendered tile delivery | -| N-D cubes, for example `time × y × x × band` | [GeoZarr](../zarr/intro.qmd), [COG](../cloud-optimized-geotiffs/intro.qmd) | GeoZarr for analysis across the cube; COG per time step if slices are read alone | +| N-D cubes, for example `time × y × x × band` | [GeoZarr](../zarr/intro.qmd), [COG](../cloud-optimized-geotiffs/intro.qmd), [Cloud-Optimized HDF/NetCDF](../cloud-optimized-netcdf4-hdf5/index.qmd) | GeoZarr for analysis across the cube; COG per time step if slices are read alone; cloud-optimized HDF/netCDF where an archival format is mandated or existing tooling must keep working | | Point cloud | [COPC](../copc/index.qmd), [GeoParquet](../geoparquet/index.qmd) | COPC for spatial subsetting and level of detail; GeoParquet if columnar analytics dominate | | Vector or tabular | [GeoParquet](../geoparquet/index.qmd), [FlatGeobuf](../flatgeobuf/intro.qmd), [PMTiles](../pmtiles/intro.qmd) | GeoParquet for joins and analytics; FlatGeobuf for streaming download; PMTiles for tiled web delivery | From f5d684bb875b83591736de37f054e3f12577cdcd Mon Sep 17 00:00:00 2001 From: Emmanuel Mathot Date: Mon, 10 Aug 2026 09:35:21 +0200 Subject: [PATCH 21/59] Clarify processing intent in decision framework by enhancing descriptions of on-the-fly and batched processing, improving guidance on layout needs for visualization and data fusion. --- decision-framework/index.qmd | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/decision-framework/index.qmd b/decision-framework/index.qmd index 5ced9764..406e2cfc 100644 --- a/decision-framework/index.qmd +++ b/decision-framework/index.qmd @@ -191,7 +191,7 @@ Apply these within the candidate set your topology opened. What are the data for? Archive and download, processing, visualization, distribution, or data fusion. Three points matter: - **Visualization is a first-class intent**, not a side effect. It has its own layout needs, mainly [overviews](../glossary.qmd#overviews) or a level of detail. -- **On-the-fly and batched processing pull the layout in opposite directions.** On-the-fly access rewards small chunks, fine overviews, and cache locality. Batched processing rewards large shards or row groups and a layout that reads well in sequence. +- **On-the-fly and batched processing pull the layout in opposite directions.** On-the-fly access rewards small chunks, fine overviews, and cache locality: a reader fetches a little and wants it back fast. Batched processing rewards large shards or row groups, so that a bulk scan becomes a few large contiguous reads that many workers can run in parallel, instead of thousands of small scattered ones. - **Data fusion raises the stakes on encoding.** Combining products from different missions or processors means every input must be read back as the same physical value, and subsets from each must line up. Fusion is a form of processing, but it is worth naming separately: it weights [interpretation encoding](#interpretation-encoding) and consistent grids far higher than single-product analysis does, and it usually rewards a tabular or columnar target where joins are cheap. Most real datasets serve several intents at once. Weigh them in proportion rather than picking one. The [case studies](case-studies.qmd) state an intent mix for each product, which is what decides the ranking inside a candidate set. From d9ac010b752930d8a34a42d022831ab7f8b0a057 Mon Sep 17 00:00:00 2001 From: Emmanuel Mathot Date: Mon, 10 Aug 2026 09:36:10 +0200 Subject: [PATCH 22/59] Update decision-framework/index.qmd Co-authored-by: Alex I. Mandel --- decision-framework/index.qmd | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/decision-framework/index.qmd b/decision-framework/index.qmd index 406e2cfc..560e7e53 100644 --- a/decision-framework/index.qmd +++ b/decision-framework/index.qmd @@ -289,7 +289,7 @@ These are not decision points. They apply whatever you choose. ### Metadata and cataloging -Metadata that a reader can fetch in a single request is part of [P1](#preconditions), not an extra. Every candidate the [topology question](#topology) opens has its own mechanism for it: Zarr consolidated metadata, COG headers, GeoParquet footers, and the COPC header. What this section adds is the level *above* a single file. +Metadata that a reader can fetch in a single request is part of [Precondition P1](#preconditions), not an extra. Every candidate the [topology question](#topology) opens has its own mechanism for it: Zarr consolidated metadata, COG headers, GeoParquet footers, and the COPC header. What this section adds is the level *above* a single file. At a scale of tens of thousands to millions of files, collection-level discovery becomes critical. [STAC](https://stacspec.org/) has an essential role there, and it is part of the cloud-native geospatial ecosystem that we recommend. It is **not a format and not a decision point in this framework**, so this page does not prescribe it. Choose a catalog approach that fits your organization. From 5bd680d57c37744e49ca1ddb63bbf1945017cada Mon Sep 17 00:00:00 2001 From: Emmanuel Mathot Date: Mon, 10 Aug 2026 09:39:54 +0200 Subject: [PATCH 23/59] link --- decision-framework/case-studies.qmd | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/decision-framework/case-studies.qmd b/decision-framework/case-studies.qmd index 93a2d8ce..3546ac99 100644 --- a/decision-framework/case-studies.qmd +++ b/decision-framework/case-studies.qmd @@ -7,7 +7,8 @@ Each case study below runs a real product through the [decision framework](index Read each case in two parts: -1. **The path through the spine**: the control fork, the [topology question](index.qmd#topology), then intent, the object-size lever, maturity, and interpretation encoding. +1. **The path through the spine**: the [control fork](index.qmd#control), the [topology question](index.qmd#topology), then the four refinements: [intent](index.qmd#intent), the [object-size and grouping lever](index.qmd#grouping-lever), [ecosystem maturity](index.qmd#maturity), and [interpretation encoding](index.qmd#interpretation-encoding). + 2. **What the measurement showed.** The examples use public Copernicus and NASA missions. The [CNES Datalake](https://cnes.fr/en/projects/centre-de-calcul) appears as one worked example of a tiered object store. It is not the only kind of backend, and its tier numbers are not a general rule. From 94bdb553845a560c83f40291dbe3f816f28bf86f Mon Sep 17 00:00:00 2001 From: Emmanuel Mathot Date: Mon, 10 Aug 2026 09:41:01 +0200 Subject: [PATCH 24/59] Update decision-framework/case-studies.qmd Co-authored-by: Alex I. Mandel --- decision-framework/case-studies.qmd | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/decision-framework/case-studies.qmd b/decision-framework/case-studies.qmd index 3546ac99..17f703b0 100644 --- a/decision-framework/case-studies.qmd +++ b/decision-framework/case-studies.qmd @@ -9,7 +9,7 @@ Read each case in two parts: 1. **The path through the spine**: the [control fork](index.qmd#control), the [topology question](index.qmd#topology), then the four refinements: [intent](index.qmd#intent), the [object-size and grouping lever](index.qmd#grouping-lever), [ecosystem maturity](index.qmd#maturity), and [interpretation encoding](index.qmd#interpretation-encoding). -2. **What the measurement showed.** +2. **What the benchmark measurements showed.** The examples use public Copernicus and NASA missions. The [CNES Datalake](https://cnes.fr/en/projects/centre-de-calcul) appears as one worked example of a tiered object store. It is not the only kind of backend, and its tier numbers are not a general rule. From b2d7ebff993c20e85c0e502fe02d7a8c5c981550 Mon Sep 17 00:00:00 2001 From: Emmanuel Mathot Date: Mon, 10 Aug 2026 09:53:35 +0200 Subject: [PATCH 25/59] Enhance decision framework documentation by adding access pattern definitions and clarifying its impact on chunking and storage tiers, improving user understanding of data handling. --- decision-framework/case-studies.qmd | 4 +++- decision-framework/index.qmd | 20 +++++++++++++++++++- glossary.qmd | 12 ++++++++++++ 3 files changed, 34 insertions(+), 2 deletions(-) diff --git a/decision-framework/case-studies.qmd b/decision-framework/case-studies.qmd index 17f703b0..bcbc98d6 100644 --- a/decision-framework/case-studies.qmd +++ b/decision-framework/case-studies.qmd @@ -5,13 +5,15 @@ subtitle: "Six products run through the framework, five of them backed by measur Each case study below runs a real product through the [decision framework](index.qmd). It shows where the product lands and why. Five of the six are backed by **measured benchmark results**. +If a term here is unfamiliar, the framework page defines the framework's own vocabulary in [Key terms](index.qmd#key-terms), and the guide's [glossary](../glossary.qmd) covers general cloud-native terms. + Read each case in two parts: 1. **The path through the spine**: the [control fork](index.qmd#control), the [topology question](index.qmd#topology), then the four refinements: [intent](index.qmd#intent), the [object-size and grouping lever](index.qmd#grouping-lever), [ecosystem maturity](index.qmd#maturity), and [interpretation encoding](index.qmd#interpretation-encoding). 2. **What the benchmark measurements showed.** -The examples use public Copernicus and NASA missions. The [CNES Datalake](https://cnes.fr/en/projects/centre-de-calcul) appears as one worked example of a tiered object store. It is not the only kind of backend, and its tier numbers are not a general rule. +The examples use public Copernicus and NASA missions. The [CNES Datalake](https://cnes.fr/en/projects/centre-de-calcul) appears as one worked example of a [tiered object store](../glossary.qmd#tiered-storage). It is not the only kind of backend, and its tier numbers are not a general rule. ::: {.callout-important} ## The recurring lesson diff --git a/decision-framework/index.qmd b/decision-framework/index.qmd index 560e7e53..3b347f7f 100644 --- a/decision-framework/index.qmd +++ b/decision-framework/index.qmd @@ -31,7 +31,7 @@ The framework uses a few terms in a specific way. Each links to the section that | Term | What it means here | | --- | --- | -| [**Access pattern**](#grouping-lever) | How readers will actually hit the data: a time series at one point, a spatial window at one date, a full scan. Intent implies an access pattern, and your layout either serves it or fights it. | +| [**Access pattern**](../glossary.qmd#access-pattern) | How readers will actually hit the data: a time series at one point, a spatial window at one date, a full scan. Intent implies an access pattern, and your layout either serves it or fights it. Defined in the guide's glossary; the layout consequences are in [the grouping lever](#grouping-lever). | | [**Archival-format mandate**](#control) | A requirement to keep a specific archival format for preservation, such as HDF5, netCDF or GRIB, whatever cloud-optimized format you publish alongside it. | | [**Control**](#control) | Whether the choice is yours to make at all: do the data already exist, and may you reprocess them, technically and legally. This is stage 1 of the spine, and it can end in "not yet possible". | | [**Ecosystem maturity**](#maturity) | How settled a format is, on an L1 to L5 scale from experimental to de-facto standard. It answers "safe default, or bet?". | @@ -167,6 +167,10 @@ Three tests, in order: files, a codec no parser can pass through, and a single uncompressed blob with no internal chunking. See [Not yet possible](#not-yet-possible). +**Passing these tests means you *can*, not that you *should* yet.** A producer may be +technically and legally free to change format and still be held by what its users' +tools expect. See [Existing users and their tools](#existing-users). + ### Topology: what shape are the data? {#topology} @@ -303,6 +307,20 @@ Some choices sit below the format and are foundational: These are mature and in scope. The per-format pages cover them. More experimental optimizations, such as bit-rounding and advanced compression tricks, are out of scope here, or should be marked as immature when you use them. +### Existing users and their tools {#existing-users} + +Being able to reprocess is not the same as being free to change format. If a long-standing user community reads your product with established tools, changing format breaks them, and that cost is real even when nothing technical or legal stops you. + +This resembles an [archival-format mandate](#control) but behaves differently: + +| | Archival-format mandate | User and tooling expectations | +| --- | --- | --- | +| Comes from | Institutional policy | Your consumer community | +| Negotiable | Rarely | Yes, with notice | +| Ends when | Never, you satisfy it indefinitely | You finish a migration you announced | + +In the short term the answer is usually the same: publish the cloud-optimized format alongside what your users already consume. The endpoint differs. A mandate you keep satisfying; an expectation you migrate out of, with a deprecation window and documentation. Weigh it together with [ecosystem maturity](#maturity), because the newer your target format, the more of your users' toolchain you are asking them to replace. + ### Not yet possible {#not-yet-possible} When a source cannot be repackaged or virtualized today, the honest answer is *"not yet implemented"*, not *"impossible"*. The blocker is usually a missing parser, an unsupported codec, or source files with no internal chunking at all. Every one of those can change. diff --git a/glossary.qmd b/glossary.qmd index c4c2069d..72124f51 100644 --- a/glossary.qmd +++ b/glossary.qmd @@ -9,6 +9,12 @@ format: This glossary aims to describe all the jargon of geospatial in the cloud! Are we missing something? [Create an issue](https://github.com/cloudnativegeo/cloud-optimized-geospatial-formats-guide/issues/new?title=Glossary+suggestion:&labels=glossary) to suggest an improvement. +#### Access pattern + +How readers actually reach the data: a time series at a single point, a spatial window at a single date, or a full scan of everything. + +The access pattern drives the [chunk](#chunk) shape and the order of the dimensions, which is a separate question from [chunk size](#chunk-size). A cube chunked long in time answers a time series at one point cheaply and a full map expensively; chunked wide in latitude and longitude it does the opposite. Only rewriting the data lets you change it, so a virtualized dataset inherits whatever access pattern its source was written for. + #### Amazon S3 (S3) The [object storage](#object-storage-cloud-storage) service offered by Amazon. Part of [Amazon Web Services](#amazon-web-services-aws). @@ -328,6 +334,12 @@ For further reading: [A dive into spatial search algorithms](https://blog.mapbox A file format for image and raster data that supports [lossless compression](#lossless-compression). +#### Tiered storage + +A storage service that keeps data on different classes of hardware at different prices and speeds. A common arrangement is a "warm" tier on disk for data that is read often, and a "cold" tier on tape or archival storage for data that is read rarely. + +Some providers set a minimum mean [object](#object-storage-cloud-storage) size for each tier, because indexing very many small objects is expensive. A layout whose objects fall below that floor can be refused or moved to a slower tier, so [chunk size](#chunk-size) and grouping choices have to be made against the policy of the storage you are actually using. + #### Vector data A type of geospatial data to represent points, lines, and polygons. From cba755551f1b67dea41d5cad4c96bca67929704f Mon Sep 17 00:00:00 2001 From: Emmanuel Mathot Date: Mon, 10 Aug 2026 09:58:35 +0200 Subject: [PATCH 26/59] Refine case studies in decision framework by adding detailed intent, existing data status, and grouping lever information, enhancing clarity for users on data handling and processing choices. --- decision-framework/case-studies.qmd | 60 ++++++++++++++++++++--------- 1 file changed, 42 insertions(+), 18 deletions(-) diff --git a/decision-framework/case-studies.qmd b/decision-framework/case-studies.qmd index bcbc98d6..19dd6977 100644 --- a/decision-framework/case-studies.qmd +++ b/decision-framework/case-studies.qmd @@ -33,11 +33,15 @@ They were produced with [`cng-formats-benchmark`](https://github.com/development ## 1. Per-scene optical raster: COG vs GeoZarr for visualization and distribution -**Product:** Sentinel-2 L2A. **Topology:** 2D raster per scene. **Source format:** geoTiff in a `.zip`. **Candidate set:** `{COG, GeoZarr}`. +**Product:** Sentinel-2 L2A. **Source format:** geoTiff in a `.zip`. -**[Intent](index.qmd#intent).** Distribution and visualization. Both are served per scene, which is why this case does not need a cube. +- **Existing data?** Yes. +- **[Can you reprocess?](index.qmd#control)** Yes, the producer operates the processing chain, so the format is a free choice. +- **[What shape are the data?](index.qmd#topology)** 2D raster per scene, which opens `{COG, GeoZarr}`. +- **[Intent](index.qmd#intent).** Distribution and visualization, both served per scene. That favors COG; time-series analysis would favor a GeoZarr cube instead. +- **[Grouping lever](index.qmd#grouping-lever).** COG internal tiling, or a scene-size Zarr shard. -**The path.** Existing data, and the producer operates the processing chain, so this is a free choice. The topology opens `{COG, GeoZarr}` and the [intent](index.qmd#intent) decides between them. Per-scene **distribution and visualization** favors COG. **Time-series analysis** favors a GeoZarr cube. The [grouping lever](index.qmd#grouping-lever) is COG internal tiling, or a scene-size Zarr shard. +Both intents are served per scene, which is why this case does not need a cube at all. **Measured, one scene.** Both layouts cleared the warm tier. COG came to 2625.7 MiB in 28 objects. A scene-size GeoZarr shard came to 1975.6 MiB in 26 objects, about 25 % smaller. That gap is the **codec**, Deflate against ZSTD, not the format. Against the 2023.7 MiB delivered `.zip`, the GeoZarr store is about source size (−2 %) and the COG is larger (+30 %), because the COG carries overviews. @@ -53,11 +57,15 @@ The associated masking product, which is written as many small GeoTIFFs, is a cl ## 2. Raster time-cube: GeoZarr, with COG as the alternative {#case-time-cube} -**Product:** Sentinel-1 RTC. **Topology:** 2D raster over time. **Source format:** geoTiff in a `.zip`. **Candidate set:** `{COG, GeoZarr}`. +**Product:** Sentinel-1 RTC. **Source format:** geoTiff in a `.zip`. -**[Intent](index.qmd#intent).** Advanced analysis and visualization lead, with distribution close behind but weighted a little lower. The analysis half is what separates this case from case 1: it is asked over time, not per scene, and that is what pulls the answer to a cube. +- **Existing data?** Yes. +- **[Can you reprocess?](index.qmd#control)** Yes, so the format is a free choice. +- **[What shape are the data?](index.qmd#topology)** 2D raster over time, which opens `{COG, GeoZarr}`. +- **[Intent](index.qmd#intent).** Advanced analysis and visualization lead, with distribution close behind but weighted a little lower. +- **[Grouping lever](index.qmd#grouping-lever).** Zarr sharding, sized to the backend's preference for large objects. -**The path.** Existing data, reprocessable, so a free choice. A year of RTC over a region is naturally a `time × y × x × pol` cube. On a Zarr cube, a time-series slice or an area-of-interest subset is one range-addressable query. Across thousands of per-timestamp COGs, the same request becomes a large fan-out. Sharding reconciles fine-grained chunk access with the backend's preference for large objects. +The analysis half is what separates this case from case 1: it is asked over time, not per scene. A year of RTC over a region is naturally a `time × y × x × pol` cube, and on a Zarr cube a time-series slice or an area-of-interest subset is one range-addressable query. Across thousands of per-timestamp COGs the same request becomes a large fan-out. Sharding reconciles fine-grained chunk access with the backend's preference for large objects. **Measured, one tile and confirmed at area scale.** COG came to 728 MiB per tile and a scene-size GeoZarr shard to 545 MiB. Both cleared the cold tier. GeoZarr is again about 25 % smaller, and again this is the codec: Deflate at 3.70x against ZSTD at 4.98x. The result held over a 16-tile block, at 16498 MiB for COG and 12344 MiB for GeoZarr. Against the 496 MiB delivered `.zip` both are larger, COG by 47 % and GeoZarr by 10 %. Cloud-optimized costs storage here, and buys the range access that the opaque zip cannot give at all. @@ -71,11 +79,15 @@ Both issues are being fixed for the final campaign, so that display is compared ## 3. Point cloud: netCDF to COPC -**Product:** SWOT L2_HR Pixel Cloud. **Topology:** point cloud. **Source format:** netCDF. **Candidate set:** `{COPC, GeoParquet}`. +**Product:** SWOT L2_HR Pixel Cloud. **Source format:** netCDF. -**[Intent](index.qmd#intent).** Distribution, visualization, and data fusion. The fusion intent is the one that matters most here: these points get combined with other water products, so a reader has to recover the same physical value per point and subset the same area cheaply. That is what makes the missing spatial index in the source a real cost rather than an inconvenience. +- **Existing data?** Yes. +- **[Can you reprocess?](index.qmd#control)** Yes, so the format is a free choice. +- **[What shape are the data?](index.qmd#topology)** Point cloud, which opens `{COPC, GeoParquet}`. +- **[Intent](index.qmd#intent).** Distribution, visualization, and data fusion. +- **[Grouping lever](index.qmd#grouping-lever).** COPC octree nodes; each granule already clears the tier on its own. -**The path.** Existing data, reprocessable, so a free choice. The point-cloud topology gives COPC, which is a clustered octree inside a LAZ file. The netCDF source has no spatial index and no level of detail, so reading a small geographic subset means scanning the whole granule. +The fusion intent matters most here: these points get combined with other water products, so a reader has to recover the same physical value per point and subset the same area cheaply. That is what makes the source's missing spatial index a real cost rather than an inconvenience. COPC is a clustered octree inside a LAZ file; the netCDF source has no spatial index and no level of detail, so reading a small geographic subset means scanning the whole granule. **Measured, content-complete, three granules.** A COPC carrying all per-point variables was about 16 % **smaller** than the source netCDF. This is the one clear size win among the cases, and it comes from the codec: LASzip at 3.72x against netCDF Deflate at 2.56x. Each granule cleared the tier on its own. @@ -92,11 +104,15 @@ An earlier measurement on this product looked like a 97 % size saving. It had dr ## 4. Vector and tabular: shapefile to GeoParquet -**Product:** SWOT L2_HR Lake Single Pass. **Topology:** vector and tabular. **Source format:** shapefile. **Candidate set:** `{GeoParquet, FlatGeobuf}`. +**Product:** SWOT L2_HR Lake Single Pass. **Source format:** shapefile. -**[Intent](index.qmd#intent).** Distribution, download, and data fusion. Note that visualization is *absent* from this mix, and download is present. That combination is why the answer is not automatically the columnar option: download rewards a single streamable file, which is FlatGeobuf's strength, while fusion rewards cheap joins on a fixed schema, which is GeoParquet's. The fusion weight is what settles it. +- **Existing data?** Yes. +- **[Can you reprocess?](index.qmd#control)** Yes, so the format is a free choice. +- **[What shape are the data?](index.qmd#topology)** Vector and tabular, which opens `{GeoParquet, FlatGeobuf}`. +- **[Intent](index.qmd#intent).** Distribution, download, and data fusion. Visualization is *absent* from this mix. +- **[Grouping lever](index.qmd#grouping-lever).** Row groups, but only after aggregating across passes; a single pass is below the warm-tier floor. -**The path.** Existing data, reprocessable, so a free choice. This is a fixed-schema tabular product, with a lake id, a reach id, water-surface elevation, area, and more. That is the native fit for GeoParquet, with its columns, row groups, and spatial partitioning. A sparse Zarr store would add complexity with no gain. +The intent mix is what makes this case interesting: visualization is absent and download is present, so the answer is not automatically the columnar one. Download rewards a single streamable file, which is FlatGeobuf's strength, while fusion rewards cheap joins on a fixed schema, which is GeoParquet's. The fusion weight settles it. The product is fixed-schema tabular, with a lake id, a reach id, water-surface elevation, area and more, which is the native fit for GeoParquet's columns, row groups and spatial partitioning. A sparse Zarr store would add complexity with no gain. **Measured, two passes.** GeoParquet came to 17.8 MiB per pass, about 14 % **larger** than the 31.1 MiB zipped shapefile. This surprised reviewers, who expected a binary columnar format to be smaller. The likely cause is the default row-group codec, and it is under investigation. @@ -106,11 +122,13 @@ Each pass sits below the warm-tier floor. The lever is therefore **aggregation a ## 5. Small N-D grid: the aggregation regime -**Product:** SWOT L2_HR Raster. **Topology:** N-D grid. **Source format:** geoTiff. **Candidate set:** `{GeoZarr, COG}`. +**Product:** SWOT L2_HR Raster. **Source format:** geoTiff. -**[Intent](index.qmd#intent).** Distribution and download, with data fusion against the other SWOT products, and some visualization. This mix is provisional and still to be confirmed with the product owner. - -**The path.** Existing data, reprocessable, so a free choice. A 100 m raster granule covers a small tile, about 160 km across. +- **Existing data?** Yes. +- **[Can you reprocess?](index.qmd#control)** Yes, so the format is a free choice. +- **[What shape are the data?](index.qmd#topology)** N-D grid, which opens `{GeoZarr, COG}`. +- **[Intent](index.qmd#intent).** Distribution and download, with data fusion against the other SWOT products, and some visualization. This mix is provisional and still to be confirmed with the product owner. +- **[Grouping lever](index.qmd#grouping-lever).** Aggregation *above* the product, not sharding within it. A 100 m granule covers a small tile, about 160 km across, so it is far below the warm-tier floor. **Measured, single variable, so not yet like for like.** On the primary water-surface-elevation variable, COG came to 4.3 MiB per granule and a GeoZarr shard to 0.8 MiB on average, because sharding fragments at this size. Both sit far below the warm-tier floor, so the lever is **aggregation**, and at this small size **COG is the cleaner carrier**: one tiled object that also holds overviews. An all-variable re-run is needed before quoting a size, for the same reason as case 3. @@ -122,9 +140,15 @@ This case is the counterpart to cases 1 and 2. There, one large band already cle ## 6. Existing archive to a virtual store -**Topology:** any. **Candidate set:** a virtual Zarr store. +**Product:** any immutable archive, for example HDF5 or netCDF. **Source format:** whatever was delivered. + +- **Existing data?** Yes. +- **[Can you reprocess?](index.qmd#control)** No. +- **Can you repackage the delivered files?** No. +- **[Virtualizable?](index.qmd#control)** Yes, so a virtual Zarr store is the candidate. +- **[Grouping lever](index.qmd#grouping-lever).** None available. You inherit the source's chunking exactly. -This case has no benchmark. It illustrates the "existing data, cannot reprocess" branch of the control fork. +This case has no benchmark. It illustrates the "existing data, cannot reprocess" branch of the control fork, and it is the only case that does not reach the topology question at all. An immutable archive, for example HDF5 or netCDF that you cannot reprocess, can be surfaced as a virtual Zarr store with [VirtualiZarr, Kerchunk, or Icechunk](../zarr/virtual-zarr.qmd). This gives range-based access without rewriting the data. From f4e3bf32b94715299a08639b1aee5b3d396cf4e5 Mon Sep 17 00:00:00 2001 From: Emmanuel Mathot Date: Mon, 10 Aug 2026 10:05:26 +0200 Subject: [PATCH 27/59] Clarify the precedent of HDF5 in the decision framework by detailing improvements in reader tooling and performance metrics, enhancing user understanding of format evolution and library capabilities. --- decision-framework/index.qmd | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/decision-framework/index.qmd b/decision-framework/index.qmd index 3b347f7f..2fe1d0d0 100644 --- a/decision-framework/index.qmd +++ b/decision-framework/index.qmd @@ -325,7 +325,7 @@ In the short term the answer is usually the same: publish the cloud-optimized fo When a source cannot be repackaged or virtualized today, the honest answer is *"not yet implemented"*, not *"impossible"*. The blocker is usually a missing parser, an unsupported codec, or source files with no internal chunking at all. Every one of those can change. -The precedent is HDF5 itself: once the standard example of a format that could not be read efficiently from object storage, until a change in write strategy and reader tooling made it viable, see [Cloud-Optimized HDF/NetCDF](../cloud-optimized-netcdf4-hdf5/index.qmd). Formats move. +The precedent is HDF5 itself: once the standard example of a format that could not be read efficiently from object storage, until a change in write strategy and reader tooling made it viable, see [Cloud-Optimized HDF/NetCDF](../cloud-optimized-netcdf4-hdf5/index.qmd).The reader half is still moving: switching xarray's default cache for the h5netcdf engine improves dataset opening ([xarray #11216](https://github.com/pydata/xarray/pull/11216), measured by its authors on NISAR data). Formats move, and so do the libraries. Frame the branch that way, and route the reader back to the options: reprocess if you can get the chain changed, virtualize if a reader exists, or convert on ingest. Say what is missing, so somebody can unlock it. From 45d16ca93e50174c7f8785f446b3363c6efb7d84 Mon Sep 17 00:00:00 2001 From: Emmanuel Mathot Date: Mon, 10 Aug 2026 10:11:26 +0200 Subject: [PATCH 28/59] Enhance case study on spatial indexing by clarifying the three-dimensional aspect of the octree and its impact on bounding box queries, improving user understanding of data access and measurement relevance. --- decision-framework/case-studies.qmd | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/decision-framework/case-studies.qmd b/decision-framework/case-studies.qmd index 19dd6977..d23fc103 100644 --- a/decision-framework/case-studies.qmd +++ b/decision-framework/case-studies.qmd @@ -93,14 +93,14 @@ The fusion intent matters most here: these points get combined with other water The real gain is what the netCDF lacks: -- A **spatial index**. A bounding-box query reads only the overlapping octree nodes. +- A **spatial index in three dimensions**. The octree subdivides in x, y *and* z, so a query can constrain a bounding box **and** an elevation range, reading only the overlapping nodes. That matters here because water surface elevation is the measurement, not merely a coordinate. - A **level of detail**. Octree levels 0 to 11 are the point-cloud analogue of COG overviews. ::: {.callout-caution} An earlier measurement on this product looked like a 97 % size saving. It had dropped most of the per-point variables. Size comparisons must be like for like on content. ::: -**Where it lands.** COPC. The cloud-optimized value is partial access plus level of detail, with a modest size win on top. GeoParquet is the alternative where columnar analytics dominate. +**Where it lands.** COPC. The cloud-optimized value is partial access plus level of detail, with a modest size win on top. GeoParquet is the alternative where columnar analytics dominate, though 3D pruning there is optional: the [spec](https://geoparquet.org/releases/v1.1.0/) allows `zmin` and `zmax` in the bbox covering, but the writer has to populate them and the engine has to use them. ## 4. Vector and tabular: shapefile to GeoParquet @@ -162,7 +162,7 @@ This case is also where the ["not yet possible"](index.qmd#not-yet-possible) fra | --- | --- | --- | --- | --- | | 1. Optical scene | 2D raster | COG per scene; a cube needs cross-scene aggregation | COG +30 %, GeoZarr −2 % | Partial access, encoding | | 2. SAR time-cube | Raster over time | GeoZarr v3 sharded | COG +47 %, GeoZarr +10 % | One query instead of a fan-out | -| 3. Pixel cloud | Point cloud | COPC | −16 % | Spatial index, level of detail | +| 3. Pixel cloud | Point cloud | COPC | −16 % | 3D spatial index, level of detail | | 4. Lake vector | Vector, tabular | GeoParquet, aggregated | +14 % | Row-group bbox index | | 5. Small raster grid | N-D grid | COG, plus aggregation | Re-run pending | Fewer, larger objects | | 6. Frozen archive | Any | Virtual Zarr | No rewrite | Range access with no reprocessing | From e7fc113d64a8deef0c4cbdfba5c4b162e6d88971 Mon Sep 17 00:00:00 2001 From: Emmanuel Mathot Date: Mon, 10 Aug 2026 10:29:22 +0200 Subject: [PATCH 29/59] Refine case study on GeoParquet and FlatGeobuf by clarifying download and fusion advantages, enhancing user understanding of format performance and access patterns. --- decision-framework/case-studies.qmd | 16 +++++++++++++--- 1 file changed, 13 insertions(+), 3 deletions(-) diff --git a/decision-framework/case-studies.qmd b/decision-framework/case-studies.qmd index d23fc103..ff6066c8 100644 --- a/decision-framework/case-studies.qmd +++ b/decision-framework/case-studies.qmd @@ -112,13 +112,23 @@ An earlier measurement on this product looked like a 97 % size saving. It had dr - **[Intent](index.qmd#intent).** Distribution, download, and data fusion. Visualization is *absent* from this mix. - **[Grouping lever](index.qmd#grouping-lever).** Row groups, but only after aggregating across passes; a single pass is below the warm-tier floor. -The intent mix is what makes this case interesting: visualization is absent and download is present, so the answer is not automatically the columnar one. Download rewards a single streamable file, which is FlatGeobuf's strength, while fusion rewards cheap joins on a fixed schema, which is GeoParquet's. The fusion weight settles it. The product is fixed-schema tabular, with a lake id, a reach id, water-surface elevation, area and more, which is the native fit for GeoParquet's columns, row groups and spatial partitioning. A sparse Zarr store would add complexity with no gain. +The intent mix makes this case a genuine two-way contest, though not in the way it first appears: -**Measured, two passes.** GeoParquet came to 17.8 MiB per pass, about 14 % **larger** than the 31.1 MiB zipped shapefile. This surprised reviewers, who expected a binary columnar format to be smaller. The likely cause is the default row-group codec, and it is under investigation. +- **Download favors GeoParquet, not FlatGeobuf.** FlatGeobuf has [no internal compression](../flatgeobuf/intro.qmd#no-internal-compression), and it cannot simply be gzipped, because its spatial index addresses byte ranges in the *uncompressed* file. GeoParquet [compresses per column by default](../geoparquet/index.qmd#internal-compression-by-default), so within this candidate set it is the smaller transfer. +- **Streaming is a visualization property, not a download one.** FlatGeobuf [streams](../flatgeobuf/intro.qmd#streaming-features-is-supported), so features render progressively as they arrive. Valuable, but visualization is absent from this mix. +- **FlatGeobuf wins on whole-row spatial subsetting.** It is row-oriented with a Hilbert R-tree, so every attribute of the features in a bounding box is one contiguous read, where a [column-oriented](../geoparquet/index.qmd#column-oriented) store has to touch every column. A real access pattern, just not one this intent mix asks for. + +At volume the scale question would matter too, because GeoParquet shards across [multiple files](../geoparquet/index.qmd#multi-file-support) while FlatGeobuf is a single file with a single index. These passes are tiny, so it does not bite here. + +So GeoParquet wins on download *and* fusion. The product is fixed-schema tabular, with a lake id, a reach id, water-surface elevation, area and more, which is the native fit for its columns and row groups. A sparse Zarr store would add complexity with no gain. + +**Measured, two passes.** GeoParquet came to 17.8 MiB per pass, so 35.6 MiB for the two passes, against 31.1 MiB for the same two passes as a zipped shapefile. That is about 14 % **larger**, and the ratio is the same per pass (17.8 against 15.6 MiB). + +Note that this compares GeoParquet against the **source**, not against FlatGeobuf. Against its actual rival in the candidate set, GeoParquet is the smaller of the two, because FlatGeobuf cannot compress at all. Each pass sits below the warm-tier floor. The lever is therefore **aggregation across passes**: concatenate first, then size the row groups. A one-for-one transcode of each pass would leave the object-size problem in place. The gain over the shapefile is the bounding-box row-group index, not the size. -**Where it lands.** GeoParquet, with pass-level aggregation. FlatGeobuf is the single-file streaming alternative. +**Where it lands.** GeoParquet, with pass-level aggregation. FlatGeobuf is the alternative where whole-row spatial subsetting or progressive rendering dominates, and where the data stays small enough for a single file. ## 5. Small N-D grid: the aggregation regime From 754932e21e073ff6fa16facd13769fd4fbfaa886 Mon Sep 17 00:00:00 2001 From: Emmanuel Mathot Date: Mon, 10 Aug 2026 10:30:48 +0200 Subject: [PATCH 30/59] Update decision-framework/index.qmd Co-authored-by: Aimee Barciauskas --- decision-framework/index.qmd | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/decision-framework/index.qmd b/decision-framework/index.qmd index 2fe1d0d0..92bd32b7 100644 --- a/decision-framework/index.qmd +++ b/decision-framework/index.qmd @@ -11,7 +11,7 @@ It is a guide, not an oracle: - The **shape of your data** narrows the field. - The **intended use** and a few practical levers rank the remaining options. -- **You keep the final call.** +- **You make the final call.** The framework is provider-neutral. Where a concrete number helps, such as an object-size policy or a benchmark result, the number is given as a worked example and flagged as an example, not as a rule. From d8bc1ac424fd675884ed3321ce7fdb49b36feee5 Mon Sep 17 00:00:00 2001 From: Emmanuel Mathot Date: Mon, 10 Aug 2026 10:31:00 +0200 Subject: [PATCH 31/59] Update decision-framework/case-studies.qmd Co-authored-by: Aimee Barciauskas --- decision-framework/case-studies.qmd | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/decision-framework/case-studies.qmd b/decision-framework/case-studies.qmd index ff6066c8..0da6b81a 100644 --- a/decision-framework/case-studies.qmd +++ b/decision-framework/case-studies.qmd @@ -16,7 +16,7 @@ Read each case in two parts: The examples use public Copernicus and NASA missions. The [CNES Datalake](https://cnes.fr/en/projects/centre-de-calcul) appears as one worked example of a [tiered object store](../glossary.qmd#tiered-storage). It is not the only kind of backend, and its tier numbers are not a general rule. ::: {.callout-important} -## The recurring lesson +## Cloud-optimized != smaller storage size **Going cloud-optimized is rarely smaller than the source you started from.** From 35ecb16e46c19e305f32ee7e3d6f1cb6a2a48470 Mon Sep 17 00:00:00 2001 From: Emmanuel Mathot Date: Mon, 10 Aug 2026 10:31:21 +0200 Subject: [PATCH 32/59] Update decision-framework/case-studies.qmd Co-authored-by: Aimee Barciauskas --- decision-framework/case-studies.qmd | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/decision-framework/case-studies.qmd b/decision-framework/case-studies.qmd index 0da6b81a..b9d99152 100644 --- a/decision-framework/case-studies.qmd +++ b/decision-framework/case-studies.qmd @@ -20,7 +20,7 @@ The examples use public Copernicus and NASA missions. The [CNES Datalake](https: **Going cloud-optimized is rarely smaller than the source you started from.** -What it buys you is partial access, a spatial index, or a level of detail. Where two cloud-optimized candidates *do* differ in size, the cause is usually the **codec**, not the format. +Cloud-optimized formats provide partial access via byte-range indices into the data, accessible via addressable metadata. A second element explains much of the gap: **the extra bytes are the feature.** Overviews, levels of detail, and spatial indexes are structures the source file did not carry at all. They add volume, and they are exactly what buys fast display and cheap subsetting. The COG in case 1 is about 30 % larger than the delivered archive mostly because it carries overviews, and the row-group index in case 4 is the whole point of the conversion. ::: From c58814f07ed6afea68b5d46dffc02fe1faa197aa Mon Sep 17 00:00:00 2001 From: Emmanuel Mathot Date: Mon, 10 Aug 2026 10:35:04 +0200 Subject: [PATCH 33/59] Clarify product description and virtualization conditions for large immutable archives, enhancing understanding of chunked array formats and their limitations in the decision framework. --- decision-framework/case-studies.qmd | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/decision-framework/case-studies.qmd b/decision-framework/case-studies.qmd index b9d99152..739ad12f 100644 --- a/decision-framework/case-studies.qmd +++ b/decision-framework/case-studies.qmd @@ -150,12 +150,12 @@ This case is the counterpart to cases 1 and 2. There, one large band already cle ## 6. Existing archive to a virtual store -**Product:** any immutable archive, for example HDF5 or netCDF. **Source format:** whatever was delivered. +**Product:** a large immutable archive in a chunked array format, for example HDF5 or netCDF. **Source format:** whatever was delivered. - **Existing data?** Yes. - **[Can you reprocess?](index.qmd#control)** No. - **Can you repackage the delivered files?** No. -- **[Virtualizable?](index.qmd#control)** Yes, so a virtual Zarr store is the candidate. +- **[Virtualizable?](index.qmd#control)** Yes, *if* the source is a chunked array format. Kerchunk and VirtualiZarr cover netCDF4, HDF5, GRIB2, TIFF and FITS. Most vector formats have no equivalent, so this branch is effectively raster and N-D only. - **[Grouping lever](index.qmd#grouping-lever).** None available. You inherit the source's chunking exactly. This case has no benchmark. It illustrates the "existing data, cannot reprocess" branch of the control fork, and it is the only case that does not reach the topology question at all. @@ -164,6 +164,8 @@ An immutable archive, for example HDF5 or netCDF that you cannot reprocess, can The result is more limited than a native store. Chunk sizes are whatever the source chose, you cannot add overviews, and you inherit the source's codec. It is still much better than whole-file download, and it is often the only option available. +**"Cannot reprocess" usually means "too big".** This branch is normally reached for reasons of scale rather than permission: the archive is large enough that duplicating it, or spending the compute to regenerate it, costs more than the access problem is worth. That is why it is mostly an array and raster story. A vector dataset is usually small enough that [repackaging](index.qmd#control) it outright is cheaper than indexing around it, so vector products rarely land here. + This case is also where the ["not yet possible"](index.qmd#not-yet-possible) framing matters. If the source cannot even be virtualized, name what is missing rather than declaring the case impossible. ## Summary @@ -175,7 +177,7 @@ This case is also where the ["not yet possible"](index.qmd#not-yet-possible) fra | 3. Pixel cloud | Point cloud | COPC | −16 % | 3D spatial index, level of detail | | 4. Lake vector | Vector, tabular | GeoParquet, aggregated | +14 % | Row-group bbox index | | 5. Small raster grid | N-D grid | COG, plus aggregation | Re-run pending | Fewer, larger objects | -| 6. Frozen archive | Any | Virtual Zarr | No rewrite | Range access with no reprocessing | +| 6. Frozen archive | Chunked arrays only | Virtual Zarr | No rewrite | Range access with no reprocessing | Read the size column together with the recurring lesson at the top of this page. Only one case got smaller, and that case got smaller because of its codec. From 55aeb645730e857ef0c34b3e326e09c235e0200b Mon Sep 17 00:00:00 2001 From: Emmanuel Mathot Date: Mon, 10 Aug 2026 10:38:16 +0200 Subject: [PATCH 34/59] Add clarification on data control factors in decision framework --- decision-framework/index.qmd | 1 + 1 file changed, 1 insertion(+) diff --git a/decision-framework/index.qmd b/decision-framework/index.qmd index 92bd32b7..d40c8aa4 100644 --- a/decision-framework/index.qmd +++ b/decision-framework/index.qmd @@ -9,6 +9,7 @@ This framework helps you go from *"I have geospatial data to publish or migrate" It is a guide, not an oracle: +- **What you control** comes first: whether the data already exist, whether you may reprocess them, and whether a mandate fixes an archival format. This can settle the answer before format even comes up. - The **shape of your data** narrows the field. - The **intended use** and a few practical levers rank the remaining options. - **You make the final call.** From 79187c06b2253416dc7a85a62ade8970ad1c8ce2 Mon Sep 17 00:00:00 2001 From: Emmanuel Mathot Date: Mon, 10 Aug 2026 10:41:22 +0200 Subject: [PATCH 35/59] Refine definition of "Access pattern" for clarity in glossary and decision framework --- decision-framework/index.qmd | 2 +- glossary.qmd | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/decision-framework/index.qmd b/decision-framework/index.qmd index d40c8aa4..c230fee2 100644 --- a/decision-framework/index.qmd +++ b/decision-framework/index.qmd @@ -32,7 +32,7 @@ The framework uses a few terms in a specific way. Each links to the section that | Term | What it means here | | --- | --- | -| [**Access pattern**](../glossary.qmd#access-pattern) | How readers will actually hit the data: a time series at one point, a spatial window at one date, a full scan. Intent implies an access pattern, and your layout either serves it or fights it. Defined in the guide's glossary; the layout consequences are in [the grouping lever](#grouping-lever). | +| [**Access pattern**](../glossary.qmd#access-pattern) | What subsets readers will request: a time series at one point, a spatial window at one date, a full scan. Intent implies an access pattern, and your layout either serves it or fights it. Defined in the guide's glossary; the layout consequences are in [the grouping lever](#grouping-lever). | | [**Archival-format mandate**](#control) | A requirement to keep a specific archival format for preservation, such as HDF5, netCDF or GRIB, whatever cloud-optimized format you publish alongside it. | | [**Control**](#control) | Whether the choice is yours to make at all: do the data already exist, and may you reprocess them, technically and legally. This is stage 1 of the spine, and it can end in "not yet possible". | | [**Ecosystem maturity**](#maturity) | How settled a format is, on an L1 to L5 scale from experimental to de-facto standard. It answers "safe default, or bet?". | diff --git a/glossary.qmd b/glossary.qmd index 72124f51..89aeafec 100644 --- a/glossary.qmd +++ b/glossary.qmd @@ -11,7 +11,7 @@ This glossary aims to describe all the jargon of geospatial in the cloud! Are we #### Access pattern -How readers actually reach the data: a time series at a single point, a spatial window at a single date, or a full scan of everything. +What subsets of the data get requested: a time series at a single point, a spatial window at a single date, or a full scan of everything. The access pattern drives the [chunk](#chunk) shape and the order of the dimensions, which is a separate question from [chunk size](#chunk-size). A cube chunked long in time answers a time series at one point cheaply and a full map expensively; chunked wide in latitude and longitude it does the opposite. Only rewriting the data lets you change it, so a virtualized dataset inherits whatever access pattern its source was written for. From 01e395b81a0648ec799319cb813b8efb55df317b Mon Sep 17 00:00:00 2001 From: Emmanuel Mathot Date: Mon, 10 Aug 2026 10:41:52 +0200 Subject: [PATCH 36/59] Update decision-framework/case-studies.qmd Co-authored-by: Aimee Barciauskas --- decision-framework/case-studies.qmd | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/decision-framework/case-studies.qmd b/decision-framework/case-studies.qmd index 739ad12f..4d1d6775 100644 --- a/decision-framework/case-studies.qmd +++ b/decision-framework/case-studies.qmd @@ -18,7 +18,7 @@ The examples use public Copernicus and NASA missions. The [CNES Datalake](https: ::: {.callout-important} ## Cloud-optimized != smaller storage size -**Going cloud-optimized is rarely smaller than the source you started from.** +**A cloud-optimized format is not necessarily smaller than a non-cloud-optimized version.** Cloud-optimized formats provide partial access via byte-range indices into the data, accessible via addressable metadata. From eb6b7c39a89e97f051e4beab7c9c1e6c9d24de92 Mon Sep 17 00:00:00 2001 From: Emmanuel Mathot Date: Mon, 10 Aug 2026 10:42:07 +0200 Subject: [PATCH 37/59] Update decision-framework/case-studies.qmd Co-authored-by: Aimee Barciauskas --- decision-framework/case-studies.qmd | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/decision-framework/case-studies.qmd b/decision-framework/case-studies.qmd index 4d1d6775..81737558 100644 --- a/decision-framework/case-studies.qmd +++ b/decision-framework/case-studies.qmd @@ -22,7 +22,7 @@ The examples use public Copernicus and NASA missions. The [CNES Datalake](https: Cloud-optimized formats provide partial access via byte-range indices into the data, accessible via addressable metadata. -A second element explains much of the gap: **the extra bytes are the feature.** Overviews, levels of detail, and spatial indexes are structures the source file did not carry at all. They add volume, and they are exactly what buys fast display and cheap subsetting. The COG in case 1 is about 30 % larger than the delivered archive mostly because it carries overviews, and the row-group index in case 4 is the whole point of the conversion. +**The extra bytes are the feature.** Overviews, levels of detail, and spatial indexes are structures the source file did not carry at all. They add volume, and they are exactly what buys fast display and cheap subsetting. The COG in case 1 is about 30% larger than the delivered archive mostly because it carries overviews, and the row-group index in case 4 is the whole point of the conversion. ::: ::: {.callout-note} From 13820f57be7d0604dd256281f49a17dd97986111 Mon Sep 17 00:00:00 2001 From: Emmanuel Mathot Date: Mon, 10 Aug 2026 10:42:25 +0200 Subject: [PATCH 38/59] Update decision-framework/index.qmd Co-authored-by: Aimee Barciauskas --- decision-framework/index.qmd | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/decision-framework/index.qmd b/decision-framework/index.qmd index c230fee2..3617cdce 100644 --- a/decision-framework/index.qmd +++ b/decision-framework/index.qmd @@ -33,7 +33,7 @@ The framework uses a few terms in a specific way. Each links to the section that | Term | What it means here | | --- | --- | | [**Access pattern**](../glossary.qmd#access-pattern) | What subsets readers will request: a time series at one point, a spatial window at one date, a full scan. Intent implies an access pattern, and your layout either serves it or fights it. Defined in the guide's glossary; the layout consequences are in [the grouping lever](#grouping-lever). | -| [**Archival-format mandate**](#control) | A requirement to keep a specific archival format for preservation, such as HDF5, netCDF or GRIB, whatever cloud-optimized format you publish alongside it. | +| [**Archival-format mandate**](#control) | A requirement to maintain a specific archival format for preservation, such as HDF5, netCDF or GRIB, whatever cloud-optimized format you publish alongside it. | | [**Control**](#control) | Whether the choice is yours to make at all: do the data already exist, and may you reprocess them, technically and legally. This is stage 1 of the spine, and it can end in "not yet possible". | | [**Ecosystem maturity**](#maturity) | How settled a format is, on an L1 to L5 scale from experimental to de-facto standard. It answers "safe default, or bet?". | | [**Grouping lever**](#grouping-lever) | The knob each format gives you for grouping bytes into objects: COG internal tiling, Zarr v3 sharding, COPC octree nodes, GeoParquet row groups. | From f92d87ecce6e3731f9728d7ecee5db1d6815e2d2 Mon Sep 17 00:00:00 2001 From: Emmanuel Mathot Date: Mon, 10 Aug 2026 10:42:37 +0200 Subject: [PATCH 39/59] Update decision-framework/case-studies.qmd Co-authored-by: Aimee Barciauskas --- decision-framework/case-studies.qmd | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/decision-framework/case-studies.qmd b/decision-framework/case-studies.qmd index 81737558..75cc10a7 100644 --- a/decision-framework/case-studies.qmd +++ b/decision-framework/case-studies.qmd @@ -7,7 +7,13 @@ Each case study below runs a real product through the [decision framework](index If a term here is unfamiliar, the framework page defines the framework's own vocabulary in [Key terms](index.qmd#key-terms), and the guide's [glossary](../glossary.qmd) covers general cloud-native terms. -Read each case in two parts: +Each case study includes: + +1. Characteristics of the product +2. What the data is intended for +3. What path the case study follows through the decision tree +4. A size measurement for each candidate format +5. The decision outcome 1. **The path through the spine**: the [control fork](index.qmd#control), the [topology question](index.qmd#topology), then the four refinements: [intent](index.qmd#intent), the [object-size and grouping lever](index.qmd#grouping-lever), [ecosystem maturity](index.qmd#maturity), and [interpretation encoding](index.qmd#interpretation-encoding). From c66568605effdd93a1c030927b8812f835192f7c Mon Sep 17 00:00:00 2001 From: Emmanuel Mathot Date: Mon, 10 Aug 2026 10:42:52 +0200 Subject: [PATCH 40/59] Update decision-framework/index.qmd Co-authored-by: Aimee Barciauskas --- decision-framework/index.qmd | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/decision-framework/index.qmd b/decision-framework/index.qmd index 3617cdce..d2e37aa7 100644 --- a/decision-framework/index.qmd +++ b/decision-framework/index.qmd @@ -236,7 +236,7 @@ performs. And it is mostly settled back at the [control](#control) question, bec ### Ecosystem maturity {#maturity} -The real question is not a version number. It is whether you need something production-grade that you can count on, or you are willing to be an early adopter. Maturity is what separates a safe default from a bet. +Ecosystem maturity may influence the final decision. You may need something production-grade that you can count on, or you may have the flexibility to be an early adopter. Maturity is what separates a safe default from a bet. To keep that judgement explicit rather than impressionistic, place each candidate on a five-level scale: From 30bba09a0aad59d13e01dedaac20c8d7b5e9f176 Mon Sep 17 00:00:00 2001 From: Emmanuel Mathot Date: Mon, 10 Aug 2026 10:43:15 +0200 Subject: [PATCH 41/59] Update decision-framework/index.qmd Co-authored-by: Aimee Barciauskas --- decision-framework/index.qmd | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/decision-framework/index.qmd b/decision-framework/index.qmd index d2e37aa7..b20d04c5 100644 --- a/decision-framework/index.qmd +++ b/decision-framework/index.qmd @@ -262,7 +262,7 @@ Where the candidates sit today: **For GeoParquet and COPC, the risk is integration effort, not the specification.** Each inherits a mature base format and adds a spatial index on top. -#### Zarr v3 and GeoZarr are two different questions +#### New Zarr Standards: Distinguishing Zarr v3 and GeoZarr This distinction is easy to miss, and it decides how much risk you actually take on. From ad1686f6f7a4a4726b3098fcd6fd441458a63f3f Mon Sep 17 00:00:00 2001 From: Emmanuel Mathot Date: Mon, 10 Aug 2026 10:47:06 +0200 Subject: [PATCH 42/59] Clarify example of object-size policy in storage backend section --- decision-framework/index.qmd | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/decision-framework/index.qmd b/decision-framework/index.qmd index b20d04c5..74366569 100644 --- a/decision-framework/index.qmd +++ b/decision-framework/index.qmd @@ -214,7 +214,7 @@ Every candidate format has a knob that sets how bytes are grouped into objects: Tune the lever to your **storage backend's object-size policy**. [Object stores](../glossary.qmd#object-storage-cloud-storage) degrade when they must index very many tiny objects, so most backends reward a larger average object. -One worked example: the [CNES Datalake](https://cnes.fr/en/projects/centre-de-calcul) sorts data into tiers by mean object size, with about 32 MB or more for warm disk and about 100 MB or more for cold tape. Those numbers belong to that backend. Your backend will have its own policy, and it may have none. Read the tiers as an instance of a general concern. +One worked example: the [CNES Datalake](https://cnes.fr/en/projects/centre-de-calcul) sorts data into [tiers](../glossary.qmd#tiered-storage) by mean object size, with about 32 MB or more for warm disk and about 100 MB or more for cold tape. This serves the general point: find out whether your storage has an object-size policy, and size the lever against it. Some providers publish a threshold, some only charge you as if your objects were larger, and some have no policy at all. The lever works in both directions, and which direction you need depends on product size: From b9b132745ee3aa05b85095de88294550c28be392 Mon Sep 17 00:00:00 2001 From: Emmanuel Mathot Date: Mon, 10 Aug 2026 10:50:06 +0200 Subject: [PATCH 43/59] Clarify the role of shape in format selection and its impact on performance --- decision-framework/index.qmd | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/decision-framework/index.qmd b/decision-framework/index.qmd index 74366569..925cf801 100644 --- a/decision-framework/index.qmd +++ b/decision-framework/index.qmd @@ -229,9 +229,11 @@ serves a time series at one point cheaply and a full-scene map expensively; chun wide in `y` and `x` it does the opposite. This guide works it through, with a diagram, in [chunk shape vs chunk size](../cloud-optimized-netcdf4-hdf5/index.qmd#chunk-shape-vs-chunk-size). -Shape rarely decides *which* format you pick, because every member of a family -exposes some version of the knob. It decides how well the format you picked -performs. And it is mostly settled back at the [control](#control) question, because only reprocessing lets you choose the shape freely. +Shape rarely decides *which* format you pick, because every candidate in a family lets +you set it: a COG chooses its tile dimensions, Zarr its chunk shape and dimension +order, GeoParquet its row-group size and row ordering. What shape decides is how well +the format you picked performs, not which one you pick. And it is mostly settled back +at the [control](#control) question, because only reprocessing lets you choose it freely. ### Ecosystem maturity {#maturity} From 0008534e26937b0b1d39238620a8b902fc08cf4c Mon Sep 17 00:00:00 2001 From: Emmanuel Mathot Date: Mon, 10 Aug 2026 10:53:44 +0200 Subject: [PATCH 44/59] Clarify terminology by replacing "spine" with "framework" in decision framework documentation --- decision-framework/case-studies.qmd | 2 +- decision-framework/index.qmd | 10 +++++----- 2 files changed, 6 insertions(+), 6 deletions(-) diff --git a/decision-framework/case-studies.qmd b/decision-framework/case-studies.qmd index 75cc10a7..3b15148d 100644 --- a/decision-framework/case-studies.qmd +++ b/decision-framework/case-studies.qmd @@ -15,7 +15,7 @@ Each case study includes: 4. A size measurement for each candidate format 5. The decision outcome -1. **The path through the spine**: the [control fork](index.qmd#control), the [topology question](index.qmd#topology), then the four refinements: [intent](index.qmd#intent), the [object-size and grouping lever](index.qmd#grouping-lever), [ecosystem maturity](index.qmd#maturity), and [interpretation encoding](index.qmd#interpretation-encoding). +1. **The path through the framework**: the [control fork](index.qmd#control), the [topology question](index.qmd#topology), then the four refinements: [intent](index.qmd#intent), the [object-size and grouping lever](index.qmd#grouping-lever), [ecosystem maturity](index.qmd#maturity), and [interpretation encoding](index.qmd#interpretation-encoding). 2. **What the benchmark measurements showed.** diff --git a/decision-framework/index.qmd b/decision-framework/index.qmd index 925cf801..054be4ea 100644 --- a/decision-framework/index.qmd +++ b/decision-framework/index.qmd @@ -34,7 +34,7 @@ The framework uses a few terms in a specific way. Each links to the section that | --- | --- | | [**Access pattern**](../glossary.qmd#access-pattern) | What subsets readers will request: a time series at one point, a spatial window at one date, a full scan. Intent implies an access pattern, and your layout either serves it or fights it. Defined in the guide's glossary; the layout consequences are in [the grouping lever](#grouping-lever). | | [**Archival-format mandate**](#control) | A requirement to maintain a specific archival format for preservation, such as HDF5, netCDF or GRIB, whatever cloud-optimized format you publish alongside it. | -| [**Control**](#control) | Whether the choice is yours to make at all: do the data already exist, and may you reprocess them, technically and legally. This is stage 1 of the spine, and it can end in "not yet possible". | +| [**Control**](#control) | Whether the choice is yours to make at all: do the data already exist, and may you reprocess them, technically and legally. This is stage 1 of the framework, and it can end in "not yet possible". | | [**Ecosystem maturity**](#maturity) | How settled a format is, on an L1 to L5 scale from experimental to de-facto standard. It answers "safe default, or bet?". | | [**Grouping lever**](#grouping-lever) | The knob each format gives you for grouping bytes into objects: COG internal tiling, Zarr v3 sharding, COPC octree nodes, GeoParquet row groups. | | [**Intent**](#intent) | What the data are for: archive and download, processing, visualization, distribution, or data fusion. Most products serve several at once, in proportion. | @@ -53,9 +53,9 @@ The framework does not restate that definition. It assumes it. Two consequences **P2. No opaque bundles.** An [archive format](../glossary.qmd#archive-format) is not disqualified just for being an archive. [PMTiles](../pmtiles/intro.qmd) is an archive and it is cloud-optimized, because its internal directory makes each tile addressable. What disqualifies a candidate is an archive with no addressable internal layout, or a layer of [external compression](../glossary.qmd#external-compression) wrapped around data that was already cloud-optimized. A zipped Zarr store, a LAZ file without [COPC metadata](https://copc.io/), and a single Parquet file with no row groups all fail this way: the client must fetch the whole object before it can use any part of it. -## The spine +## The two stages -The spine runs in **two stages**, and the topology question is the hinge between them. +The framework runs in **two stages**, and the topology question is the hinge between them. **Stage 1 — control: do you have a choice?** Do the data already exist? If they exist, are you able (technically, legally) to reprocess them? This stage can end @@ -334,7 +334,7 @@ Frame the branch that way, and route the reader back to the options: reprocess i ## Optional: make it quantitative -The spine above gets you to a short list. When two or three candidates remain and the choice is close, a transparent score can make the trade-off explicit rather than a matter of taste. This section is **optional**. The framework stands on its own without it. +The two stages above get you to a short list. When two or three candidates remain and the choice is close, a transparent score can make the trade-off explicit rather than a matter of taste. This section is **optional**. The framework stands on its own without it. The idea is a **weighted mean of criteria**, each expressed as a number from 0 to 1. It is framework-first and metrics-optional: qualitative terms alone already give a ranking, and a benchmark run sharpens the same ranking. @@ -359,7 +359,7 @@ The convincing artifact is never a single score. It is a sensitivity sweep that ## Worked examples -Six products run through the spine on the [Case Studies](case-studies.qmd) page. Five are backed by measurements. +Six products run through the framework on the [Case Studies](case-studies.qmd) page. Five are backed by measurements. One lesson recurs often enough to state up front: **going cloud-optimized is rarely smaller than the source you started from.** Overviews, levels of detail, and spatial indexes are structures the source did not carry, so they add volume, and that added volume is the feature you are buying. Where two candidates do differ in size, the cause is usually the **codec**, not the format. From 5df3ade1cc5777aacd82a45da499a259b953998b Mon Sep 17 00:00:00 2001 From: Emmanuel Mathot Date: Mon, 10 Aug 2026 10:56:34 +0200 Subject: [PATCH 45/59] Update decision-framework/index.qmd Co-authored-by: Aimee Barciauskas --- decision-framework/index.qmd | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/decision-framework/index.qmd b/decision-framework/index.qmd index 054be4ea..5a8ee24e 100644 --- a/decision-framework/index.qmd +++ b/decision-framework/index.qmd @@ -135,7 +135,7 @@ flowchart TD **Existing data you cannot reprocess, but can rewrite.** You may still be allowed to repackage the delivered files: a plain TIFF becomes a COG, a LAZ file becomes a COPC, a shapefile becomes GeoParquet. The format is a free choice, but the content is fixed at whatever was delivered. Go to the [topology question](#topology). -**Existing data you cannot reprocess.** You can often still publish a virtual view over the bytes you have. [Virtual Zarr](../zarr/virtual-zarr.qmd), with VirtualiZarr, Kerchunk, or Icechunk, builds an index that points into the original files. This gives range-based access without rewriting the data. The result is more limited than a native store, but it is much better than nothing. +**Existing data you cannot reprocess.** You can often still publish a virtual view over the bytes you have. The [Virtual Zarr](../zarr/virtual-zarr.qmd) approach is to use a library (such as VirtualiZarr or Kerchunk) to build an index that points into the original files. That index is then stored in formats such as kerchunk json, kerchunk parquet or Icechunk. This gives range-based access without rewriting the data. The result is more limited than a native store, but it is much better than nothing. **Neither.** Some sources cannot be repackaged or virtualized today. Examples are HDF4, encrypted files, and a single large uncompressed blob. See [Not yet possible](#not-yet-possible). From afb32a0d816b3b93b1ba932bc2dd21579fcae48b Mon Sep 17 00:00:00 2001 From: Emmanuel Mathot Date: Mon, 10 Aug 2026 10:58:44 +0200 Subject: [PATCH 46/59] Clarify terminology by replacing "backend" with "object store" in decision framework documentation --- decision-framework/case-studies.qmd | 6 +++--- decision-framework/index.qmd | 10 +++++----- 2 files changed, 8 insertions(+), 8 deletions(-) diff --git a/decision-framework/case-studies.qmd b/decision-framework/case-studies.qmd index 3b15148d..9274b0c7 100644 --- a/decision-framework/case-studies.qmd +++ b/decision-framework/case-studies.qmd @@ -19,7 +19,7 @@ Each case study includes: 2. **What the benchmark measurements showed.** -The examples use public Copernicus and NASA missions. The [CNES Datalake](https://cnes.fr/en/projects/centre-de-calcul) appears as one worked example of a [tiered object store](../glossary.qmd#tiered-storage). It is not the only kind of backend, and its tier numbers are not a general rule. +The examples use public Copernicus and NASA missions. The [CNES Datalake](https://cnes.fr/en/projects/centre-de-calcul) appears as one worked example of a [tiered object store](../glossary.qmd#tiered-storage). It is not the only kind of object store, and its tier numbers are not a general rule. ::: {.callout-important} ## Cloud-optimized != smaller storage size @@ -69,9 +69,9 @@ The associated masking product, which is written as many small GeoTIFFs, is a cl - **[Can you reprocess?](index.qmd#control)** Yes, so the format is a free choice. - **[What shape are the data?](index.qmd#topology)** 2D raster over time, which opens `{COG, GeoZarr}`. - **[Intent](index.qmd#intent).** Advanced analysis and visualization lead, with distribution close behind but weighted a little lower. -- **[Grouping lever](index.qmd#grouping-lever).** Zarr sharding, sized to the backend's preference for large objects. +- **[Grouping lever](index.qmd#grouping-lever).** Zarr sharding, sized to the object store's preference for large objects. -The analysis half is what separates this case from case 1: it is asked over time, not per scene. A year of RTC over a region is naturally a `time × y × x × pol` cube, and on a Zarr cube a time-series slice or an area-of-interest subset is one range-addressable query. Across thousands of per-timestamp COGs the same request becomes a large fan-out. Sharding reconciles fine-grained chunk access with the backend's preference for large objects. +The analysis half is what separates this case from case 1: it is asked over time, not per scene. A year of RTC over a region is naturally a `time × y × x × pol` cube, and on a Zarr cube a time-series slice or an area-of-interest subset is one range-addressable query. Across thousands of per-timestamp COGs the same request becomes a large fan-out. Sharding reconciles fine-grained chunk access with the object store's preference for large objects. **Measured, one tile and confirmed at area scale.** COG came to 728 MiB per tile and a scene-size GeoZarr shard to 545 MiB. Both cleared the cold tier. GeoZarr is again about 25 % smaller, and again this is the codec: Deflate at 3.70x against ZSTD at 4.98x. The result held over a 16-tile block, at 16498 MiB for COG and 12344 MiB for GeoZarr. Against the 496 MiB delivered `.zip` both are larger, COG by 47 % and GeoZarr by 10 %. Cloud-optimized costs storage here, and buys the range access that the opaque zip cannot give at all. diff --git a/decision-framework/index.qmd b/decision-framework/index.qmd index 5a8ee24e..e007f8de 100644 --- a/decision-framework/index.qmd +++ b/decision-framework/index.qmd @@ -39,7 +39,7 @@ The framework uses a few terms in a specific way. Each links to the section that | [**Grouping lever**](#grouping-lever) | The knob each format gives you for grouping bytes into objects: COG internal tiling, Zarr v3 sharding, COPC octree nodes, GeoParquet row groups. | | [**Intent**](#intent) | What the data are for: archive and download, processing, visualization, distribution, or data fusion. Most products serve several at once, in proportion. | | [**Interpretation encoding**](#interpretation-encoding) | Whether the format carries units, CRS, nodata and any scale and offset transform in-format, so every reader recovers the same physical value, or leans on a side catalog. | -| [**Object size**](#grouping-lever) | The size of the objects your layout actually writes to storage. Object stores reward a larger average object, and many backends apply a size policy or storage tier. | +| [**Object size**](#grouping-lever) | The size of the objects your layout actually writes to storage. Object stores reward a larger average object, and many apply a minimum size policy or sort objects into storage tiers. | | [**Refinement**](#refinements) | A criterion applied *inside* a format family to rank its members. There are four: intent, the grouping lever, ecosystem maturity, and interpretation encoding. | | [**Topology**](#topology) | The shape of the data: 2D raster, N-D cubes, point cloud, or vector and tabular. Answering *what shape are the data?* selects a format family, not a format. | @@ -212,7 +212,7 @@ Every candidate format has a knob that sets how bytes are grouped into objects: | COPC | octree nodes | | GeoParquet | row groups, plus partitioning | -Tune the lever to your **storage backend's object-size policy**. [Object stores](../glossary.qmd#object-storage-cloud-storage) degrade when they must index very many tiny objects, so most backends reward a larger average object. +Tune the lever to your [object store's](../glossary.qmd#object-storage-cloud-storage) **object-size policy** (S3, Azure Blob Storage, Google Cloud Storage, or an on-premises equivalent). Object stores degrade when they must index very many tiny objects, so most reward a larger average object. One worked example: the [CNES Datalake](https://cnes.fr/en/projects/centre-de-calcul) sorts data into [tiers](../glossary.qmd#tiered-storage) by mean object size, with about 32 MB or more for warm disk and about 100 MB or more for cold tape. This serves the general point: find out whether your storage has an object-size policy, and size the lever against it. Some providers publish a threshold, some only charge you as if your objects were larger, and some have no policy at all. @@ -222,9 +222,9 @@ The lever works in both directions, and which direction you need depends on prod - **Products that are small.** Sharding within one granule only makes the fragmentation worse. Group *above* the product instead: mosaic or concatenate many granules into one range-addressable object. **Size and shape are two different questions.** How *big* the objects are is a -storage-backend question, answered above. How the bytes are arranged *inside* them +question for your object store, answered above. How the bytes are arranged *inside* them (chunk shape, and the order of the dimensions) is an **access-pattern** question, -and the backend has no opinion on it. A `time × y × x` cube chunked long in time +and the object store has no opinion on it. A `time × y × x` cube chunked long in time serves a time series at one point cheaply and a full-scene map expensively; chunked wide in `y` and `x` it does the opposite. This guide works it through, with a diagram, in [chunk shape vs chunk size](../cloud-optimized-netcdf4-hdf5/index.qmd#chunk-shape-vs-chunk-size). @@ -352,7 +352,7 @@ Rank the candidates and read the **margin**, which is the gap to the runner-up. Two points are easy to get wrong: -- **Object-size and tier fit is reported information, not a scored penalty.** Report whether a layout clears a given tier, the mean and median object size, and a small-file warning **beside** the score, never inside it. A storage backend's policy is a property of that backend, not of the format, so you must weigh it against your own infrastructure. +- **Object-size and tier fit is reported information, not a scored penalty.** Report whether a layout clears a given tier, the mean and median object size, and a small-file warning **beside** the score, never inside it. An object store's policy is a property of that store, not of the format, so you must weigh it against your own infrastructure. - **Intent weights are your input.** The suitability matrix is a shared expert reference on each format's strengths, and it is useful on its own. The intent mix and the criterion weights belong to your context. The convincing artifact is never a single score. It is a sensitivity sweep that shows the ranking holds across a range of weights. That is also the honest reason this section stays short and links out: a score you cannot reproduce on your own data is not worth much, so the harness matters more than any number this page could print. From 1323fa3483fa4299898f37c9a4e2086b912643af Mon Sep 17 00:00:00 2001 From: Emmanuel Mathot Date: Mon, 10 Aug 2026 10:59:28 +0200 Subject: [PATCH 47/59] Update decision-framework/index.qmd Co-authored-by: Aimee Barciauskas --- decision-framework/index.qmd | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/decision-framework/index.qmd b/decision-framework/index.qmd index e007f8de..4501e319 100644 --- a/decision-framework/index.qmd +++ b/decision-framework/index.qmd @@ -141,7 +141,7 @@ flowchart TD #### How to tell which branch you are on -"Can you reprocess?" and "can you virtualize?" have concrete tests, not judgement calls. +Whether you can reprocess or viftualize usually depends on the data and data project context. Answers to these questions are not judgement calls. **You can reprocess** if you have the processor, the inputs, and the budget to re-run the production chain. Then format and layout are both free choices: go to From de216a8511c65e8db63dd556bcd352c73e914d21 Mon Sep 17 00:00:00 2001 From: Emmanuel Mathot Date: Mon, 10 Aug 2026 10:59:45 +0200 Subject: [PATCH 48/59] Update decision-framework/case-studies.qmd Co-authored-by: Aimee Barciauskas --- decision-framework/case-studies.qmd | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/decision-framework/case-studies.qmd b/decision-framework/case-studies.qmd index 9274b0c7..2dab304c 100644 --- a/decision-framework/case-studies.qmd +++ b/decision-framework/case-studies.qmd @@ -21,7 +21,7 @@ Each case study includes: The examples use public Copernicus and NASA missions. The [CNES Datalake](https://cnes.fr/en/projects/centre-de-calcul) appears as one worked example of a [tiered object store](../glossary.qmd#tiered-storage). It is not the only kind of object store, and its tier numbers are not a general rule. -::: {.callout-important} +::: {.callout-note} ## Cloud-optimized != smaller storage size **A cloud-optimized format is not necessarily smaller than a non-cloud-optimized version.** From 315dcc23adea5af273090a768392399ac06a4fcc Mon Sep 17 00:00:00 2001 From: Emmanuel Mathot Date: Mon, 10 Aug 2026 11:02:26 +0200 Subject: [PATCH 49/59] Clarify benchmark examples and terminology in case studies section --- decision-framework/case-studies.qmd | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/decision-framework/case-studies.qmd b/decision-framework/case-studies.qmd index 2dab304c..988f4f00 100644 --- a/decision-framework/case-studies.qmd +++ b/decision-framework/case-studies.qmd @@ -19,7 +19,7 @@ Each case study includes: 2. **What the benchmark measurements showed.** -The examples use public Copernicus and NASA missions. The [CNES Datalake](https://cnes.fr/en/projects/centre-de-calcul) appears as one worked example of a [tiered object store](../glossary.qmd#tiered-storage). It is not the only kind of object store, and its tier numbers are not a general rule. +The examples use public Copernicus and NASA missions. The [CNES Datalake](https://cnes.fr/en/projects/centre-de-calcul) appears as one worked example of a [tiered object store](../glossary.qmd#tiered-storage). Its two thresholds appear throughout the cases below: a **warm tier** with a mean-object-size floor of about 32 MB on disk, and a **cold tier** of about 100 MB on tape. It is not the only kind of object store, and its tier numbers are not a general rule. ::: {.callout-note} ## Cloud-optimized != smaller storage size @@ -49,7 +49,7 @@ They were produced with [`cng-formats-benchmark`](https://github.com/development Both intents are served per scene, which is why this case does not need a cube at all. -**Measured, one scene.** Both layouts cleared the warm tier. COG came to 2625.7 MiB in 28 objects. A scene-size GeoZarr shard came to 1975.6 MiB in 26 objects, about 25 % smaller. That gap is the **codec**, Deflate against ZSTD, not the format. Against the 2023.7 MiB delivered `.zip`, the GeoZarr store is about source size (−2 %) and the COG is larger (+30 %), because the COG carries overviews. +**Measured, one scene.** Both layouts cleared the [warm tier](../glossary.qmd#tiered-storage). COG came to 2625.7 MiB in 28 objects. A scene-size GeoZarr shard came to 1975.6 MiB in 26 objects, about 25 % smaller. That gap is the **codec**, Deflate against ZSTD, not the format. Against the 2023.7 MiB delivered `.zip`, the GeoZarr store is about source size (−2 %) and the COG is larger (+30 %), because the COG carries overviews. **[Interpretation encoding](index.qmd#interpretation-encoding) is a real differentiator here.** The source reflectance is stored as scaled integers, and the nodata fill is left undeclared. COG puts the scale, the offset, and the nodata in side metadata that readers honor unevenly. Zarr v3 can carry all three as first-class array properties, using the `scale_offset` codec and a declared fill value, so every reader recovers the same physical value. @@ -73,7 +73,7 @@ The associated masking product, which is written as many small GeoTIFFs, is a cl The analysis half is what separates this case from case 1: it is asked over time, not per scene. A year of RTC over a region is naturally a `time × y × x × pol` cube, and on a Zarr cube a time-series slice or an area-of-interest subset is one range-addressable query. Across thousands of per-timestamp COGs the same request becomes a large fan-out. Sharding reconciles fine-grained chunk access with the object store's preference for large objects. -**Measured, one tile and confirmed at area scale.** COG came to 728 MiB per tile and a scene-size GeoZarr shard to 545 MiB. Both cleared the cold tier. GeoZarr is again about 25 % smaller, and again this is the codec: Deflate at 3.70x against ZSTD at 4.98x. The result held over a 16-tile block, at 16498 MiB for COG and 12344 MiB for GeoZarr. Against the 496 MiB delivered `.zip` both are larger, COG by 47 % and GeoZarr by 10 %. Cloud-optimized costs storage here, and buys the range access that the opaque zip cannot give at all. +**Measured, one tile and confirmed at area scale.** COG came to 728 MiB per tile and a scene-size GeoZarr shard to 545 MiB. Both cleared the [cold tier](../glossary.qmd#tiered-storage). GeoZarr is again about 25 % smaller, and again this is the codec: Deflate at 3.70x against ZSTD at 4.98x. The result held over a 16-tile block, at 16498 MiB for COG and 12344 MiB for GeoZarr. Against the 496 MiB delivered `.zip` both are larger, COG by 47 % and GeoZarr by 10 %. Cloud-optimized costs storage here, and buys the range access that the opaque zip cannot give at all. ::: {.callout-warning} ## A benchmark caveat, not a format verdict From d142f07a62a6542cdcc9398cef816cbf155ac3aa Mon Sep 17 00:00:00 2001 From: Emmanuel Mathot Date: Mon, 10 Aug 2026 11:04:01 +0200 Subject: [PATCH 50/59] Clarify GeoZarr store description and timing numbers in case studies section --- decision-framework/case-studies.qmd | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/decision-framework/case-studies.qmd b/decision-framework/case-studies.qmd index 988f4f00..12be82af 100644 --- a/decision-framework/case-studies.qmd +++ b/decision-framework/case-studies.qmd @@ -55,7 +55,7 @@ Both intents are served per scene, which is why this case does not need a cube a **Where it lands.** COG per scene for distribution and visualization. -For temporal analysis GeoZarr is the right target, but **the arm measured here does not yet deliver it.** What was written is a scene-size store: one scene, so there is no time dimension to exploit. It proves the container and the encoding, because it comes out at roughly source size and it carries the scale, offset, and nodata correctly. A time-series query against it still has to open one store per date, which is the same fan-out the COG layout has. +For temporal analysis GeoZarr is the right target, but **the GeoZarr store measured here does not yet deliver it.** What was written is a scene-size store: one scene, so there is no time dimension to exploit. It proves the container and the encoding, because it comes out at roughly source size and it carries the scale, offset, and nodata correctly. A time-series query against it still has to open one store per date, which is the same fan-out the COG layout has. The temporal benefit appears only once many scenes are concatenated along time into a single cube. That is the regime measured in [case 2](#case-time-cube), and it is a collection-level layout decision, not something a scene-size store can give you. Read the per-scene store as a building block and the cube as the product. @@ -80,7 +80,7 @@ The analysis half is what separates this case from case 1: it is asked over time The GeoZarr store read and displayed much slower than the COG, but it was written **with no multiscale overview pyramid**. A store without multiscales is not a faithful GeoZarr, and the missing pyramid is the reason a tile server has to downsample full-resolution chunks on request. The slow read is also in large part a reader effect, from the library used rather than the format. -Both issues are being fixed for the final campaign, so that display is compared on equal terms. Treat the timing numbers from this arm as provisional. +Both issues are being fixed for the final campaign, so that display is compared on equal terms. Treat these timing numbers as provisional. ::: ## 3. Point cloud: netCDF to COPC From 8e04630a10b77a3f1c6f4a3846b9946e7f243928 Mon Sep 17 00:00:00 2001 From: Emmanuel Mathot Date: Mon, 10 Aug 2026 11:07:14 +0200 Subject: [PATCH 51/59] Remove benchmark caveat from GeoZarr case study for clarity --- decision-framework/case-studies.qmd | 8 -------- 1 file changed, 8 deletions(-) diff --git a/decision-framework/case-studies.qmd b/decision-framework/case-studies.qmd index 12be82af..4a3e0364 100644 --- a/decision-framework/case-studies.qmd +++ b/decision-framework/case-studies.qmd @@ -75,14 +75,6 @@ The analysis half is what separates this case from case 1: it is asked over time **Measured, one tile and confirmed at area scale.** COG came to 728 MiB per tile and a scene-size GeoZarr shard to 545 MiB. Both cleared the [cold tier](../glossary.qmd#tiered-storage). GeoZarr is again about 25 % smaller, and again this is the codec: Deflate at 3.70x against ZSTD at 4.98x. The result held over a 16-tile block, at 16498 MiB for COG and 12344 MiB for GeoZarr. Against the 496 MiB delivered `.zip` both are larger, COG by 47 % and GeoZarr by 10 %. Cloud-optimized costs storage here, and buys the range access that the opaque zip cannot give at all. -::: {.callout-warning} -## A benchmark caveat, not a format verdict - -The GeoZarr store read and displayed much slower than the COG, but it was written **with no multiscale overview pyramid**. A store without multiscales is not a faithful GeoZarr, and the missing pyramid is the reason a tile server has to downsample full-resolution chunks on request. The slow read is also in large part a reader effect, from the library used rather than the format. - -Both issues are being fixed for the final campaign, so that display is compared on equal terms. Treat these timing numbers as provisional. -::: - ## 3. Point cloud: netCDF to COPC **Product:** SWOT L2_HR Pixel Cloud. **Source format:** netCDF. From 3405a29046139db3ee41bb177d824876f92ca298 Mon Sep 17 00:00:00 2001 From: Emmanuel Mathot Date: Fri, 21 Aug 2026 09:18:27 +0200 Subject: [PATCH 52/59] Update decision-framework/index.qmd Co-authored-by: Alex I. Mandel --- decision-framework/index.qmd | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/decision-framework/index.qmd b/decision-framework/index.qmd index 4501e319..79d810fd 100644 --- a/decision-framework/index.qmd +++ b/decision-framework/index.qmd @@ -42,7 +42,7 @@ The framework uses a few terms in a specific way. Each links to the section that | [**Object size**](#grouping-lever) | The size of the objects your layout actually writes to storage. Object stores reward a larger average object, and many apply a minimum size policy or sort objects into storage tiers. | | [**Refinement**](#refinements) | A criterion applied *inside* a format family to rank its members. There are four: intent, the grouping lever, ecosystem maturity, and interpretation encoding. | | [**Topology**](#topology) | The shape of the data: 2D raster, N-D cubes, point cloud, or vector and tabular. Answering *what shape are the data?* selects a format family, not a format. | - +: {tbl-colwidths="[15,85]"} ## Preconditions This guide already defines what makes a format cloud-optimized. The Welcome page states it as a four-part [cloud-optimized format pattern](../index.qmd#the-opportunity), and the glossary defines [cloud-optimized](../glossary.qmd#cloud-optimized) as the ability to read a meaningful *part* of a file without downloading *all* of it. From ae41b3972709a741a3f8941fc478556dec3e44dc Mon Sep 17 00:00:00 2001 From: Emmanuel Mathot Date: Thu, 27 Aug 2026 09:09:30 +0200 Subject: [PATCH 53/59] Add clarification on Sentinel-2 multiscale dimensions and enhance terminology in decision framework --- decision-framework/case-studies.qmd | 2 ++ decision-framework/index.qmd | 46 +++++++++++++++++++---------- 2 files changed, 32 insertions(+), 16 deletions(-) diff --git a/decision-framework/case-studies.qmd b/decision-framework/case-studies.qmd index 4a3e0364..ff31e4dd 100644 --- a/decision-framework/case-studies.qmd +++ b/decision-framework/case-studies.qmd @@ -61,6 +61,8 @@ The temporal benefit appears only once many scenes are concatenated along time i The associated masking product, which is written as many small GeoTIFFs, is a clear anti-pattern to fix at the source. +Sentinel-2 also has a second multiscale dimension worth naming: its bands are natively captured at 10 m, 20 m, and 60 m, not derived from each other by downsampling. See [Multiscale and native mixed resolution](index.qmd#multiscale-and-native-mixed-resolution) for how one aligned pyramid can host all three without picking a single working resolution. + ## 2. Raster time-cube: GeoZarr, with COG as the alternative {#case-time-cube} **Product:** Sentinel-1 RTC. **Source format:** geoTiff in a `.zip`. diff --git a/decision-framework/index.qmd b/decision-framework/index.qmd index 79d810fd..e178cd68 100644 --- a/decision-framework/index.qmd +++ b/decision-framework/index.qmd @@ -34,7 +34,7 @@ The framework uses a few terms in a specific way. Each links to the section that | --- | --- | | [**Access pattern**](../glossary.qmd#access-pattern) | What subsets readers will request: a time series at one point, a spatial window at one date, a full scan. Intent implies an access pattern, and your layout either serves it or fights it. Defined in the guide's glossary; the layout consequences are in [the grouping lever](#grouping-lever). | | [**Archival-format mandate**](#control) | A requirement to maintain a specific archival format for preservation, such as HDF5, netCDF or GRIB, whatever cloud-optimized format you publish alongside it. | -| [**Control**](#control) | Whether the choice is yours to make at all: do the data already exist, and may you reprocess them, technically and legally. This is stage 1 of the framework, and it can end in "not yet possible". | +| [**Control**](#control) | Whether the choice is yours to make at all: do the data already exist, and may you reprocess them, technically and legally. This is Stage 1 of the framework, and it can end in "not yet possible". | | [**Ecosystem maturity**](#maturity) | How settled a format is, on an L1 to L5 scale from experimental to de-facto standard. It answers "safe default, or bet?". | | [**Grouping lever**](#grouping-lever) | The knob each format gives you for grouping bytes into objects: COG internal tiling, Zarr v3 sharding, COPC octree nodes, GeoParquet row groups. | | [**Intent**](#intent) | What the data are for: archive and download, processing, visualization, distribution, or data fusion. Most products serve several at once, in proportion. | @@ -43,6 +43,7 @@ The framework uses a few terms in a specific way. Each links to the section that | [**Refinement**](#refinements) | A criterion applied *inside* a format family to rank its members. There are four: intent, the grouping lever, ecosystem maturity, and interpretation encoding. | | [**Topology**](#topology) | The shape of the data: 2D raster, N-D cubes, point cloud, or vector and tabular. Answering *what shape are the data?* selects a format family, not a format. | : {tbl-colwidths="[15,85]"} + ## Preconditions This guide already defines what makes a format cloud-optimized. The Welcome page states it as a four-part [cloud-optimized format pattern](../index.qmd#the-opportunity), and the glossary defines [cloud-optimized](../glossary.qmd#cloud-optimized) as the ability to read a meaningful *part* of a file without downloading *all* of it. @@ -55,7 +56,7 @@ The framework does not restate that definition. It assumes it. Two consequences ## The two stages -The framework runs in **two stages**, and the topology question is the hinge between them. +The framework runs in **two stages**, and the topology question marks the hand-off between them. **Stage 1 — control: do you have a choice?** Do the data already exist? If they exist, are you able (technically, legally) to reprocess them? This stage can end @@ -63,7 +64,7 @@ the process. If you cannot reprocess the data and cannot virtualize them, there is no format decision to make yet. **Stage 2 — topology and refinement: which format?** You reach this stage only -when stage 1 leaves you free to choose. Here the shape of the data selects a +when Stage 1 leaves you free to choose. Here the shape of the data selects a **format family**: the small group of cloud-optimized formats built for that shape. A *family*, not a format. Two or three candidates normally come through, and the refinements below rank them. @@ -129,29 +130,32 @@ flowchart TD **No existing data.** You are designing a new product, so you can choose freely. Go to the [topology question](#topology). -**If an archival-format mandate applies,** it adds a second job, it does not remove that freedom. If policy requires you to preserve a particular format like HDF5, netCDF, GRIB, or something specific to your domain, you still publish a cloud-optimized copy for readers, and you additionally make the mandated format as cloud-friendly as it can be. Where the mandated format is HDF5 or netCDF, this guide has [specific guidance](../cloud-optimized-netcdf4-hdf5/index.qmd) on consolidated metadata and chunk sizing. For other archival formats the principle holds even where the guide has no page: optimize what you must keep, and publish what people will actually read. +**If an archival-format mandate applies,** it adds a second job, and it does not remove your freedom to choose how readers get access. Applying cloud-optimized practice to the archive itself is not optional: whatever the mandated format, make it as range-request-friendly as it can be. What is genuinely optional is what you do beyond that. The stronger, recommended option is to also publish a full cloud-optimized copy for readers, since a native copy gives real range access rather than the archive's own chunking. A lighter option is to skip the second copy and instead wrap the optimized archive in a [virtual Zarr](../zarr/virtual-zarr.qmd) layer: cheaper to maintain, but you inherit the archive's chunk shape, the same limit as [existing data you cannot reprocess](#control) below. Where the mandated format is HDF5 or netCDF, this guide has [specific guidance](../cloud-optimized-netcdf4-hdf5/index.qmd) on consolidated metadata and chunk sizing. For other archival formats the principle holds even where the guide has no page: optimize what you must keep, then choose deliberately how far you go for readers. **Existing data you can reprocess.** You control the production chain, so you can also choose freely. Go to the [topology question](#topology). **Existing data you cannot reprocess, but can rewrite.** You may still be allowed to repackage the delivered files: a plain TIFF becomes a COG, a LAZ file becomes a COPC, a shapefile becomes GeoParquet. The format is a free choice, but the content is fixed at whatever was delivered. Go to the [topology question](#topology). -**Existing data you cannot reprocess.** You can often still publish a virtual view over the bytes you have. The [Virtual Zarr](../zarr/virtual-zarr.qmd) approach is to use a library (such as VirtualiZarr or Kerchunk) to build an index that points into the original files. That index is then stored in formats such as kerchunk json, kerchunk parquet or Icechunk. This gives range-based access without rewriting the data. The result is more limited than a native store, but it is much better than nothing. +**Existing data you cannot reprocess.** You can often still publish a virtual view over the bytes you have. The [Virtual Zarr](../zarr/virtual-zarr.qmd) approach is to use a library (such as VirtualiZarr or Kerchunk) to build an index that points into the original files. That index is then stored in formats such as Icechunk, kerchunk json, or kerchunk parquet. This gives range-based access without rewriting the data. The result is more limited than a native store, but it is much better than nothing. -**Neither.** Some sources cannot be repackaged or virtualized today. Examples are HDF4, encrypted files, and a single large uncompressed blob. See [Not yet possible](#not-yet-possible). +**Neither.** Some sources cannot be repackaged or virtualized today. Examples are HDF4, encrypted files, and a single large blob with no internal structure, whether or not it's compressed. See [Not yet possible](#not-yet-possible). #### How to tell which branch you are on -Whether you can reprocess or viftualize usually depends on the data and data project context. Answers to these questions are not judgement calls. +Whether you can reprocess or virtualize usually depends on the data and data project context. Answers to these questions are not judgement calls. **You can reprocess** if you have the processor, the inputs, and the budget to re-run the production chain. Then format and layout are both free choices: go to [topology](#topology). **You can repackage** if you cannot re-run the chain but you may rewrite the delivered -files. A plain TIFF becomes a COG, a LAS or LAZ file becomes a COPC, a shapefile or CSV -becomes GeoParquet. Repackaging is a byte-level rewrite rather than a science -reprocessing, so it is often permitted where reprocessing is not. Format is a free -choice; the content is fixed at whatever was delivered. +files. A plain TIFF, JPEG2000, or MrSID image becomes a COG, a LAS or LAZ file becomes +a COPC, a shapefile or CSV becomes GeoParquet. The rewrite can be drastic and still +count as repackaging, not reprocessing: an HDF5 granule can become a COG, and many +single-scene COGs can become one Zarr cube, as long as the underlying values are not +recomputed. Repackaging is a byte-level rewrite rather than a science reprocessing, +so it is often permitted where reprocessing is not. Format is a free choice; the +content is fixed at whatever was delivered. **You can virtualize** if the bytes you have can be indexed as they are. [Virtual Zarr](../zarr/virtual-zarr.qmd) publishes that index without copying the data. @@ -160,15 +164,15 @@ Three tests, in order: 1. **Is the data chunked internally?** This is the hard requirement. A virtual store cannot change the source's chunk structure, so a single contiguous block virtualizes into one enormous chunk and buys you no useful partial access. -2. **Are those chunks a workable size?** You inherit both the size and the shape exactly, so the source's chunking decides which access patterns stay cheap. A store chunked for one access pattern cannot be virtualized into a store that is good for a different one. -3. **Is there a parser for the format?** Kerchunk and VirtualiZarr cover the common +2. **Are those chunks a workable size?** You mostly inherit the source's chunk size and shape, so its chunking limits which access patterns stay cheap. There is little room to make a store built for one access pattern also serve a very different one well. +3. **Is there a parser for the format?** VirtualiZarr and Kerchunk cover the common archival formats, including netCDF4, HDF5, GRIB2, TIFF and FITS. **Otherwise there is no path yet.** The usual blockers are HDF4, encrypted -files, a codec no parser can pass through, and a single uncompressed blob with no -internal chunking. See [Not yet possible](#not-yet-possible). +files, a codec no parser can pass through, and a single blob with no internal +chunking, whether or not it's compressed. See [Not yet possible](#not-yet-possible). -**Passing these tests means you *can*, not that you *should* yet.** A producer may be +**Passing these tests means you *can*, not that you *should*.** A producer may be technically and legally free to change format and still be held by what its users' tools expect. See [Existing users and their tools](#existing-users). @@ -310,6 +314,16 @@ Some choices sit below the format and are foundational: These are mature and in scope. The per-format pages cover them. More experimental optimizations, such as bit-rounding and advanced compression tricks, are out of scope here, or should be marked as immature when you use them. +Formats also differ in which data types they support, and in whether types can be mixed within one file, column, or schema. Those specifics live on each format's own page and are worth checking before committing to a candidate; they can rule one out as surely as topology or intent. + +### Multiscale and native mixed resolution + +A multiscale pyramid does not have to come from one source resolution downsampled by a fixed factor. The common case is **overviews**, also called [pyramids](../glossary.qmd#overviews): downsampled copies of the same band, generated at a uniform decimation so a reader can zoom out without reading and resampling full-resolution chunks. COG overviews, COPC's octree levels of detail, and most GeoZarr pyramids work this way, and it is already covered above under [ecosystem maturity](#maturity) and the [grouping lever](#grouping-lever). + +The same structure can also host bands that were never derived from each other and were simply captured at different native resolutions, as long as the steps between levels are chosen to land on each native resolution rather than a fixed factor. Sentinel-2 is the reference case, worked through in [case 1](case-studies.qmd) of the case studies: the 10 m, 20 m, and 60 m bands from the same sensor fold into one pyramid by aligning the decimation to each step (10 m to 20 m halves the resolution, 20 m to 60 m divides it by three), and the pyramid can continue past 60 m into ordinary overviews using the same mechanism. The [Zarr multiscales convention](https://github.com/zarr-conventions/multiscales) is built for exactly this: scale factors between levels are not required to be uniform, and its own reference examples use this Sentinel-2 layout. + +This still interacts with the [grouping lever](#grouping-lever), since each native resolution needs its own tiling or chunk shape, and with [interpretation encoding](#interpretation-encoding), since each band needs its own geotransform. What changes from the plain-overview case is that these become refinements within one multiscale structure. + ### Existing users and their tools {#existing-users} Being able to reprocess is not the same as being free to change format. If a long-standing user community reads your product with established tools, changing format breaks them, and that cost is real even when nothing technical or legal stops you. From 6b95927a9caf71fcba36fd7c0a05066349c0c639 Mon Sep 17 00:00:00 2001 From: Aimee Barciauskas Date: Tue, 8 Sep 2026 20:32:46 -0700 Subject: [PATCH 54/59] Minor wording changes --- decision-framework/case-studies.qmd | 14 ++++---- decision-framework/index.qmd | 56 ++++++++++++++--------------- 2 files changed, 35 insertions(+), 35 deletions(-) diff --git a/decision-framework/case-studies.qmd b/decision-framework/case-studies.qmd index ff31e4dd..9664ba12 100644 --- a/decision-framework/case-studies.qmd +++ b/decision-framework/case-studies.qmd @@ -28,11 +28,11 @@ The examples use public Copernicus and NASA missions. The [CNES Datalake](https: Cloud-optimized formats provide partial access via byte-range indices into the data, accessible via addressable metadata. -**The extra bytes are the feature.** Overviews, levels of detail, and spatial indexes are structures the source file did not carry at all. They add volume, and they are exactly what buys fast display and cheap subsetting. The COG in case 1 is about 30% larger than the delivered archive mostly because it carries overviews, and the row-group index in case 4 is the whole point of the conversion. +**The extra bytes are the feature.** Overviews, levels of detail, and spatial indexes are structures the non-cloud-optimized versions do not carry at all (at least not by definition). They add volume, and they are exactly what buys fast display and cheap subsetting. The COG in case 1 is about 30% larger than the delivered archive mostly because it carries overviews, and the row-group index in case 4 is the whole point of the conversion. ::: ::: {.callout-note} -The numbers below are first measurements on a representative unit per mission, meaning one scene, one tile, or a few granules. They are illustrative, not a published benchmark. Two of the six cases have re-runs pending, and each one says so. +The numbers below are first measurements on a representative unit per mission, meaning one scene, one tile, or a few granules. They are illustrative, not a robust benchmark. Two of the six cases have re-runs pending. They were produced with [`cng-formats-benchmark`](https://github.com/developmentseed/cng-formats-benchmark), an open-source harness that measures object-size distribution, write, read, and display per format. You can describe your own datasets in its config files and reproduce this shape of comparison on your own data. See [Optional: make it quantitative](index.qmd#optional-make-it-quantitative). ::: @@ -49,9 +49,9 @@ They were produced with [`cng-formats-benchmark`](https://github.com/development Both intents are served per scene, which is why this case does not need a cube at all. -**Measured, one scene.** Both layouts cleared the [warm tier](../glossary.qmd#tiered-storage). COG came to 2625.7 MiB in 28 objects. A scene-size GeoZarr shard came to 1975.6 MiB in 26 objects, about 25 % smaller. That gap is the **codec**, Deflate against ZSTD, not the format. Against the 2023.7 MiB delivered `.zip`, the GeoZarr store is about source size (−2 %) and the COG is larger (+30 %), because the COG carries overviews. +**Measured, one scene.** Both layouts cleared the [warm tier](../glossary.qmd#tiered-storage). COG came to 2625.7 MiB in 28 objects. A scene-size GeoZarr shard came to 1975.6 MiB in 26 objects, about 25 % smaller. The difference comes from the compression algorithm, not the format. ZSTD was used for GeoZarr, and deflate was used for COG compression. Against the 2023.7 MiB delivered `.zip`, the GeoZarr store is about source size (−2 %) and the COG is larger (+30 %), because the COG carries overviews. -**[Interpretation encoding](index.qmd#interpretation-encoding) is a real differentiator here.** The source reflectance is stored as scaled integers, and the nodata fill is left undeclared. COG puts the scale, the offset, and the nodata in side metadata that readers honor unevenly. Zarr v3 can carry all three as first-class array properties, using the `scale_offset` codec and a declared fill value, so every reader recovers the same physical value. +**[Interpretation encoding](index.qmd#interpretation-encoding) is a real differentiator here.** The source reflectance is stored as scaled integers, and the nodata fill is left undeclared. COG puts the scale, the offset, and the nodata in a sidecar metadata file that readers honor unevenly. Zarr v3 can carry all three as first-class array properties, using the `scale_offset` codec and a declared fill value, so every reader recovers the same physical value. **Where it lands.** COG per scene for distribution and visualization. @@ -75,7 +75,7 @@ Sentinel-2 also has a second multiscale dimension worth naming: its bands are na The analysis half is what separates this case from case 1: it is asked over time, not per scene. A year of RTC over a region is naturally a `time × y × x × pol` cube, and on a Zarr cube a time-series slice or an area-of-interest subset is one range-addressable query. Across thousands of per-timestamp COGs the same request becomes a large fan-out. Sharding reconciles fine-grained chunk access with the object store's preference for large objects. -**Measured, one tile and confirmed at area scale.** COG came to 728 MiB per tile and a scene-size GeoZarr shard to 545 MiB. Both cleared the [cold tier](../glossary.qmd#tiered-storage). GeoZarr is again about 25 % smaller, and again this is the codec: Deflate at 3.70x against ZSTD at 4.98x. The result held over a 16-tile block, at 16498 MiB for COG and 12344 MiB for GeoZarr. Against the 496 MiB delivered `.zip` both are larger, COG by 47 % and GeoZarr by 10 %. Cloud-optimized costs storage here, and buys the range access that the opaque zip cannot give at all. +**Measured, one tile and confirmed at area scale.** COG came to 728 MiB per tile and a scene-size GeoZarr shard to 545 MiB. Both cleared the [cold tier](../glossary.qmd#tiered-storage). GeoZarr is again about 25 % smaller, and again this is the codec: Deflate at 3.70x against ZSTD at 4.98x. The result held over a 16-tile block, at 16498 MiB for COG and 12344 MiB for GeoZarr. Against the 496 MiB delivered `.zip` both are larger, COG by 47 % and GeoZarr by 10 %. Cloud-optimized costs storage here but buys the range access that the opaque zip cannot give at all. ## 3. Point cloud: netCDF to COPC @@ -118,7 +118,7 @@ The intent mix makes this case a genuine two-way contest, though not in the way - **Streaming is a visualization property, not a download one.** FlatGeobuf [streams](../flatgeobuf/intro.qmd#streaming-features-is-supported), so features render progressively as they arrive. Valuable, but visualization is absent from this mix. - **FlatGeobuf wins on whole-row spatial subsetting.** It is row-oriented with a Hilbert R-tree, so every attribute of the features in a bounding box is one contiguous read, where a [column-oriented](../geoparquet/index.qmd#column-oriented) store has to touch every column. A real access pattern, just not one this intent mix asks for. -At volume the scale question would matter too, because GeoParquet shards across [multiple files](../geoparquet/index.qmd#multi-file-support) while FlatGeobuf is a single file with a single index. These passes are tiny, so it does not bite here. +At volume the scale question would matter too, because GeoParquet shards across [multiple files](../geoparquet/index.qmd#multi-file-support) while FlatGeobuf is a single file with a single index. The SWOT L2_HR passes are tiny, so the scale question does not matter here. So GeoParquet wins on download *and* fusion. The product is fixed-schema tabular, with a lake id, a reach id, water-surface elevation, area and more, which is the native fit for its columns and row groups. A sparse Zarr store would add complexity with no gain. @@ -140,7 +140,7 @@ Each pass sits below the warm-tier floor. The lever is therefore **aggregation a - **[Intent](index.qmd#intent).** Distribution and download, with data fusion against the other SWOT products, and some visualization. This mix is provisional and still to be confirmed with the product owner. - **[Grouping lever](index.qmd#grouping-lever).** Aggregation *above* the product, not sharding within it. A 100 m granule covers a small tile, about 160 km across, so it is far below the warm-tier floor. -**Measured, single variable, so not yet like for like.** On the primary water-surface-elevation variable, COG came to 4.3 MiB per granule and a GeoZarr shard to 0.8 MiB on average, because sharding fragments at this size. Both sit far below the warm-tier floor, so the lever is **aggregation**, and at this small size **COG is the cleaner carrier**: one tiled object that also holds overviews. An all-variable re-run is needed before quoting a size, for the same reason as case 3. +**Measured, single variable, so not yet like for like.** On the primary water-surface-elevation variable, COG came to 4.3 MiB per file and a GeoZarr shard to 0.8 MiB on average, because sharding fragments at this size. Both sit far below the warm-tier floor, so the lever is **aggregation**, and at this small size **COG is the cleaner carrier**: one tiled object that also holds overviews. An all-variable re-run is needed before quoting a size, for the same reason as case 3. The natural aggregation unit is a spatial mosaic, per grid zone and per orbit cycle. That turns a wall of tiny granules into one range-addressable object. diff --git a/decision-framework/index.qmd b/decision-framework/index.qmd index e178cd68..008e9634 100644 --- a/decision-framework/index.qmd +++ b/decision-framework/index.qmd @@ -10,11 +10,11 @@ This framework helps you go from *"I have geospatial data to publish or migrate" It is a guide, not an oracle: - **What you control** comes first: whether the data already exist, whether you may reprocess them, and whether a mandate fixes an archival format. This can settle the answer before format even comes up. -- The **shape of your data** narrows the field. +- The **shape of your data** narrows the set of candidate formats. - The **intended use** and a few practical levers rank the remaining options. - **You make the final call.** -The framework is provider-neutral. Where a concrete number helps, such as an object-size policy or a benchmark result, the number is given as a worked example and flagged as an example, not as a rule. +The framework is data- and cloud-provider-neutral. Within data and cloud-provider contexts, a rule may help make a decision. For example, an object-size policy or a benchmark result may influence the choice. These cases are given as worked examples, not rules. This is a first, deliberately simple version. It aligns on the questions to ask and the order to ask them in. Follow the links to a format page when you need more detail. @@ -36,10 +36,10 @@ The framework uses a few terms in a specific way. Each links to the section that | [**Archival-format mandate**](#control) | A requirement to maintain a specific archival format for preservation, such as HDF5, netCDF or GRIB, whatever cloud-optimized format you publish alongside it. | | [**Control**](#control) | Whether the choice is yours to make at all: do the data already exist, and may you reprocess them, technically and legally. This is Stage 1 of the framework, and it can end in "not yet possible". | | [**Ecosystem maturity**](#maturity) | How settled a format is, on an L1 to L5 scale from experimental to de-facto standard. It answers "safe default, or bet?". | -| [**Grouping lever**](#grouping-lever) | The knob each format gives you for grouping bytes into objects: COG internal tiling, Zarr v3 sharding, COPC octree nodes, GeoParquet row groups. | -| [**Intent**](#intent) | What the data are for: archive and download, processing, visualization, distribution, or data fusion. Most products serve several at once, in proportion. | +| [**Grouping lever**](#grouping-lever) | The knobs each format gives you for grouping bytes into data chunks and objects in cloud storage: COG internal tiling, Zarr v3 chunking and sharding, COPC octree nodes, GeoParquet row groups. | +| [**Intent**](#intent) | What the data are for: archive and download, processing, visualization, distribution, or data fusion. Most products serve several intentsat once. | | [**Interpretation encoding**](#interpretation-encoding) | Whether the format carries units, CRS, nodata and any scale and offset transform in-format, so every reader recovers the same physical value, or leans on a side catalog. | -| [**Object size**](#grouping-lever) | The size of the objects your layout actually writes to storage. Object stores reward a larger average object, and many apply a minimum size policy or sort objects into storage tiers. | +| [**Object size**](#grouping-lever) | The size of the objects your layout actually writes to storage. Object stores reward a larger average object and many apply a minimum size policy. | | [**Refinement**](#refinements) | A criterion applied *inside* a format family to rank its members. There are four: intent, the grouping lever, ecosystem maturity, and interpretation encoding. | | [**Topology**](#topology) | The shape of the data: 2D raster, N-D cubes, point cloud, or vector and tabular. Answering *what shape are the data?* selects a format family, not a format. | : {tbl-colwidths="[15,85]"} @@ -48,7 +48,7 @@ The framework uses a few terms in a specific way. Each links to the section that This guide already defines what makes a format cloud-optimized. The Welcome page states it as a four-part [cloud-optimized format pattern](../index.qmd#the-opportunity), and the glossary defines [cloud-optimized](../glossary.qmd#cloud-optimized) as the ability to read a meaningful *part* of a file without downloading *all* of it. -The framework does not restate that definition. It assumes it. Two consequences are worth naming here, because they decide whether a candidate belongs in the framework at all. They are entry conditions, not questions in the flowchart. +The framework starts from that definition Two consequences are worth naming here, because they decide whether a candidate belongs in the framework at all: **P1. The format must support partial and parallel reads over HTTP.** [Random access](../glossary.qmd#random-access) through [HTTP range requests](../glossary.qmd#http-range-request) is what makes a format usable from [object storage](../glossary.qmd#object-storage-cloud-storage). It depends on the rest of the pattern: [metadata](../glossary.qmd#metadata) in a consistent place that a reader can load in a single read, and then addressable [chunks](../glossary.qmd#chunk), internal tiling, or both. A reader that cannot locate the bytes it needs from the metadata alone cannot fetch only those bytes. @@ -66,8 +66,7 @@ is no format decision to make yet. **Stage 2 — topology and refinement: which format?** You reach this stage only when Stage 1 leaves you free to choose. Here the shape of the data selects a **format family**: the small group of cloud-optimized formats built for that -shape. A *family*, not a format. Two or three candidates normally come through, -and the refinements below rank them. +shape. The refinements below rank the options within a format family. The hand-off is worth stating outright, because the diagram can be read as if the shape of the data were the answer: @@ -142,7 +141,7 @@ flowchart TD #### How to tell which branch you are on -Whether you can reprocess or virtualize usually depends on the data and data project context. Answers to these questions are not judgement calls. +Whether you can reprocess or virtualize usually depends on the data and data project context. **You can reprocess** if you have the processor, the inputs, and the budget to re-run the production chain. Then format and layout are both free choices: go to @@ -184,7 +183,7 @@ The shape of the data selects a **format family**: the candidate set you refine | Data shape | Format family | Where each member wins | | --- | --- | --- | | 2D raster, per scene or per tile | [COG](../cloud-optimized-geotiffs/intro.qmd), [Zarr](../zarr/intro.qmd), [PMTiles](../pmtiles/intro.qmd) | COG for distribution and visualization; Zarr if the product is really a cube; PMTiles for pre-rendered tile delivery | -| N-D cubes, for example `time × y × x × band` | [GeoZarr](../zarr/intro.qmd), [COG](../cloud-optimized-geotiffs/intro.qmd), [Cloud-Optimized HDF/NetCDF](../cloud-optimized-netcdf4-hdf5/index.qmd) | GeoZarr for analysis across the cube; COG per time step if slices are read alone; cloud-optimized HDF/netCDF where an archival format is mandated or existing tooling must keep working | +| N-D cubes, for example `time × y × x × band` | [[Geo]Zarr](../zarr/intro.qmd), [COG](../cloud-optimized-geotiffs/intro.qmd), [Cloud-Optimized HDF/NetCDF](../cloud-optimized-netcdf4-hdf5/index.qmd) | GeoZarr for analysis across the cube; COG per time step if slices are read alone; cloud-optimized HDF/netCDF where an archival format is mandated or existing tooling must keep working | | Point cloud | [COPC](../copc/index.qmd), [GeoParquet](../geoparquet/index.qmd) | COPC for spatial subsetting and level of detail; GeoParquet if columnar analytics dominate | | Vector or tabular | [GeoParquet](../geoparquet/index.qmd), [FlatGeobuf](../flatgeobuf/intro.qmd), [PMTiles](../pmtiles/intro.qmd) | GeoParquet for joins and analytics; FlatGeobuf for streaming download; PMTiles for tiled web delivery | @@ -205,9 +204,9 @@ What are the data for? Archive and download, processing, visualization, distribu Most real datasets serve several intents at once. Weigh them in proportion rather than picking one. The [case studies](case-studies.qmd) state an intent mix for each product, which is what decides the ranking inside a candidate set. -### The object-size and grouping lever {#grouping-lever} +### The object size and grouping lever {#grouping-lever} -Every candidate format has a knob that sets how bytes are grouped into objects: +Every candidate format has a knob that sets how bytes are grouped into data chunks and chunks grouped into objects: | Format | Grouping lever | | --- | --- | @@ -218,12 +217,12 @@ Every candidate format has a knob that sets how bytes are grouped into objects: Tune the lever to your [object store's](../glossary.qmd#object-storage-cloud-storage) **object-size policy** (S3, Azure Blob Storage, Google Cloud Storage, or an on-premises equivalent). Object stores degrade when they must index very many tiny objects, so most reward a larger average object. -One worked example: the [CNES Datalake](https://cnes.fr/en/projects/centre-de-calcul) sorts data into [tiers](../glossary.qmd#tiered-storage) by mean object size, with about 32 MB or more for warm disk and about 100 MB or more for cold tape. This serves the general point: find out whether your storage has an object-size policy, and size the lever against it. Some providers publish a threshold, some only charge you as if your objects were larger, and some have no policy at all. +One worked example: the [CNES Datalake](https://cnes.fr/en/projects/centre-de-calcul) sorts data into [tiers](../glossary.qmd#tiered-storage) by mean object size, with about 32 MB or more for warm disk and about 100 MB or more for cold tape. This serves the general point: find out whether your storage has an object-size policy, and adjust your groupings accordingly. Some providers publish a threshold, some only charge you as if your objects were larger, and some have no policy at all. The lever works in both directions, and which direction you need depends on product size: - **Products that are already large.** Group *within* one product. Size the shard, the tile, or the row group so it clears the policy. -- **Products that are small.** Sharding within one granule only makes the fragmentation worse. Group *above* the product instead: mosaic or concatenate many granules into one range-addressable object. +- **Products that are small.** Sharding within one already small file only makes the fragmentation worse. Group *above* the product instead: mosaic or concatenate many small files into one range-addressable object. **Size and shape are two different questions.** How *big* the objects are is a question for your object store, answered above. How the bytes are arranged *inside* them @@ -244,7 +243,7 @@ at the [control](#control) question, because only reprocessing lets you choose i Ecosystem maturity may influence the final decision. You may need something production-grade that you can count on, or you may have the flexibility to be an early adopter. Maturity is what separates a safe default from a bet. -To keep that judgement explicit rather than impressionistic, place each candidate on a five-level scale: +To keep that judgement explicit rather than impressionistic, this framework assigns candidate formats on the following five-level scale: | Level | Meaning | | --- | --- | @@ -260,6 +259,7 @@ Where the candidates sit today: | --- | --- | --- | --- | | [COG](../cloud-optimized-geotiffs/intro.qmd) | De-facto standard, on a GeoTIFF base | GDAL, rio-tiler, TiTiler, every GIS | **L5** | | [GeoParquet](../geoparquet/index.qmd) | v1.x, stable, on an Apache Parquet base | GDAL, GeoPandas, DuckDB, Arrow | **L4** | +| [Zarr](../zarr/intro.qmd) | v2 and v3 stable | GDAL, zarr-python, zarrs | **L4** | | [COPC](../copc/index.qmd) | v1.0, stable, on a LAZ base | PDAL, untwine, QGIS, copc.js | **L4** | | [FlatGeobuf](../flatgeobuf/intro.qmd) | Stable, on a flatbuffers base | GDAL and OGR, QGIS, flatgeobuf.js | **L3** | | [GeoZarr](../zarr/intro.qmd) | Draft, in an OGC Standards Working Group | Emerging, mostly xarray-based stacks | **L2** | @@ -272,31 +272,31 @@ Where the candidates sit today: This distinction is easy to miss, and it decides how much risk you actually take on. -- **Zarr v3, the substrate, is dependable.** The core specification was finalized in 2024. [Sharding](#grouping-lever), the feature that matters most for object storage, is stable in `zarr-python` 3.x and readable by GDAL and xarray. Judged on its own, the Zarr v3 core behaves like an L4. +- **Zarr v3, the substrate of GeoZarr, is dependable.** The core specification was finalized in 2024. [Sharding](#grouping-lever), the feature that matters most for object storage, is stable in `zarr-python` 3.x and readable by GDAL and xarray. Judged on its own, the Zarr v3 core behaves like an L4. - **GeoZarr, the convention layer, is still being ratified.** It is what makes a generic [Zarr](../glossary.qmd#zarr) store *geospatial*: CRS, dimension semantics, and multiscale pyramids. It does for Zarr roughly what the GeoTIFF tags do for a plain TIFF. It is a draft in an OGC Standards Working Group, on the standards track but not yet an adopted OGC standard, and the published draft can lag behind active convention work. Check [geozarr.org](https://geozarr.org/) for the current status before you rely on a specific version. -So the L2 above belongs to the convention layer, not to Zarr. The leading reference implementations today, such as ESA's EOPF Sentinel output, are demonstrators rather than production guarantees. +So the L2 above belongs to the convention layer, not to Zarr. The leading reference implementations of GeoZarrtoday, such as ESA's EOPF Sentinel output, are demonstrators rather than production guarantees. #### Maturity is a warning, not a veto -An L2 candidate can still be the right answer. The N-D cubes branch is exactly that case: there is no mature alternative with the same topology, so a low level does not mean "choose something else". It means **plan for change**. Budget for convention churn, pin the versions you depend on, and expect the visualization story in particular to improve as multiscale tooling catches up. +An L2 candidate can still be the right answer. Multidimensional cubes have no mature candidate for the multidimensional topology. A low level does not mean "choose something else". It means **plan for change**. Budget for convention churn, pin the versions you depend on, and expect visualization tooling in particular to improve as tooling catches up. -Read maturity against your own tolerance. A one-off internal product and a mission archive with a twenty-year retention commitment can reasonably land on different levels from the same candidate set. +Many factors influence what maturity level you can tolerate. A one-off internal product and a mission archive with a twenty-year retention commitment can reasonably land on different levels from the same set of candidate formats. ### Interpretation encoding {#interpretation-encoding} -Does the format carry the information a reader needs to recover the same physical value, or does it lean on a side catalog? The information is units, the [CRS](../glossary.qmd#coordinate-reference-system-crs), the nodata fill, and any scale and offset transform. +Does the format carry the information a reader needs to recover the same physical value, or does it lean on a side catalog? Interpretation encoding includes units, the [CRS](../glossary.qmd#coordinate-reference-system-crs), the nodata fill value (see also [Virtualizarr's distinction of fill values](https://virtualizarr.readthedocs.io/en/latest/explanation/custom_parsers.html#fill-values)), and any scale and offset transform. -This axis separates the candidates more than people expect: +Interpretation encoding separates the candidates more than people expect: -- **Zarr v3 makes the information first-class.** With the `scale_offset` codec, a declared fill value, Zarr conventions for projection, CRS and multiscale, the transform is a first-class array property. Every reader recovers the same value. +- **Zarr v3 makes the information first-class.** With the `scale_offset` codec, a declared fill value, Zarr conventions for projection, CRS and multiscales, the transform is a first-class array property. Every reader recovers the same value. - **COG, GeoParquet, and COPC standardize the CRS** but leave the value transform to side metadata. Readers honor side metadata unevenly. -If your product stores scaled integers, or if an undeclared fill value would change a computed result, weigh this axis heavily. +If your product stores scaled integers, or if an undeclared fill value would change a computed result, weigh this aspect heavily. ## Cross-cutting considerations -These are not decision points. They apply whatever you choose. +These considerations do not impact what format you choose, but they do affect how you implement that format. ### Metadata and cataloging @@ -309,7 +309,7 @@ At a scale of tens of thousands to millions of files, collection-level discovery Some choices sit below the format and are foundational: - Base codecs, such as [ZSTD](../glossary.qmd#zstd) and [Deflate](../glossary.qmd#deflate) -- Data types and scale-offset encoding +- Data types, and scale-offset encoding - Sharding in Zarr v3 These are mature and in scope. The per-format pages cover them. More experimental optimizations, such as bit-rounding and advanced compression tricks, are out of scope here, or should be marked as immature when you use them. @@ -360,9 +360,9 @@ The idea is a **weighted mean of criteria**, each expressed as a number from 0 t | Measured, needs a benchmark run | **size**: output volume against the source | | | **performance**: measured read, write, and display, blended by the intent mix | -**Where the measured numbers come from.** The two measured criteria need a harness, and reproducibility is the whole point of quoting a number at all. [`cng-formats-benchmark`](https://github.com/developmentseed/cng-formats-benchmark) is the open-source (MIT) harness behind the numbers on the [Case Studies](case-studies.qmd) page. Per format and per dataset it measures the object-size distribution and tier fitness, write throughput, range-request-aware read latency, and tile-display latency. You describe your own datasets in config files, so no provider or dataset is baked into the code, and you run it on your own infrastructure against your own data. See the [documentation](https://developmentseed.github.io/cng-formats-benchmark/) and the [configuration guide](https://developmentseed.github.io/cng-formats-benchmark/configuration/). +**Where the measured numbers come from.** The two measured criteria need a benchmark, and reproducibility is the whole point of quoting a number at all. [`cng-formats-benchmark`](https://github.com/developmentseed/cng-formats-benchmark) is the open-source (MIT) harness behind the numbers on the [Case Studies](case-studies.qmd) page. Per format and per dataset it measures the object-size distribution and tier fitness, write throughput, range-request-aware read latency, and tile-display latency. You describe your own datasets in config files, so no provider or dataset is baked into the code, and you run it on your own infrastructure against your own data. See the [documentation](https://developmentseed.github.io/cng-formats-benchmark/) and the [configuration guide](https://developmentseed.github.io/cng-formats-benchmark/configuration/). -Rank the candidates and read the **margin**, which is the gap to the runner-up. A small margin is an honest signal that a benchmark, or a review of your weights, is what actually decides it. There are **no verdicts and no probability readout**. You get a score, a ranking, and a margin. +You can use this framework to rank the candidates and evaluate the **margin** between the winner and the runner-up. A small margin is an honest signal that a benchmark, or a review of your weights, is what actually decides it. There are **no verdicts and no probability readout**. You get a score, a ranking, and a margin. Two points are easy to get wrong: @@ -381,4 +381,4 @@ One lesson recurs often enough to state up front: **going cloud-optimized is rar This framework was developed as part of a study for the [Centre National d'Études Spatiales (CNES)](https://cnes.fr/), carried out by [Development Seed](https://developmentseed.org/). The case studies use public Copernicus and NASA missions, and the measurements come from that study's benchmark work. -The framework builds on earlier work in this guide supported by [NASA IMPACT](https://impact.earthdata.nasa.gov/). See [Our Supporters](../contributing.qmd#thank-you-to-our-supporters). +The framework builds on earlier work in this guide supported by [NASA IMPACT](https://impact.earthdata.nasa.gov/) (now [NASA ODSI](https://www.nasa.gov/marshall/marshall-space-flight-missions/office-of-data-science-and-informatics-odsi/)). See [Our Supporters](../contributing.qmd#thank-you-to-our-supporters). From 2ad4beea76b7a918a83aa2803ee0fb5d22fb466f Mon Sep 17 00:00:00 2001 From: Emmanuel Mathot Date: Wed, 9 Sep 2026 09:24:47 +0200 Subject: [PATCH 55/59] Clarify decision framework stages and object size policies in documentation --- decision-framework/case-studies.qmd | 103 +++++++++++++++++++--------- decision-framework/index.qmd | 63 ++++++++--------- glossary.qmd | 2 +- 3 files changed, 100 insertions(+), 68 deletions(-) diff --git a/decision-framework/case-studies.qmd b/decision-framework/case-studies.qmd index ff31e4dd..4f986173 100644 --- a/decision-framework/case-studies.qmd +++ b/decision-framework/case-studies.qmd @@ -9,17 +9,12 @@ If a term here is unfamiliar, the framework page defines the framework's own voc Each case study includes: -1. Characteristics of the product -2. What the data is intended for -3. What path the case study follows through the decision tree -4. A size measurement for each candidate format -5. The decision outcome +1. The characteristics of the product, and what the data is for. +2. **The path through the framework**: the [control fork](index.qmd#control), the [topology question](index.qmd#topology), then the four refinements: [intent](index.qmd#intent), the [object-size and grouping lever](index.qmd#grouping-lever), [ecosystem maturity](index.qmd#maturity), and [interpretation encoding](index.qmd#interpretation-encoding). +3. **What the benchmark measurements showed**, for each candidate format. +4. The decision outcome. -1. **The path through the framework**: the [control fork](index.qmd#control), the [topology question](index.qmd#topology), then the four refinements: [intent](index.qmd#intent), the [object-size and grouping lever](index.qmd#grouping-lever), [ecosystem maturity](index.qmd#maturity), and [interpretation encoding](index.qmd#interpretation-encoding). - -2. **What the benchmark measurements showed.** - -The examples use public Copernicus and NASA missions. The [CNES Datalake](https://cnes.fr/en/projects/centre-de-calcul) appears as one worked example of a [tiered object store](../glossary.qmd#tiered-storage). Its two thresholds appear throughout the cases below: a **warm tier** with a mean-object-size floor of about 32 MB on disk, and a **cold tier** of about 100 MB on tape. It is not the only kind of object store, and its tier numbers are not a general rule. +The examples use public Copernicus and NASA missions. The [CNES Datalake](https://cnes.fr/en/projects/centre-de-calcul) appears as one worked example of a [tiered object store](../glossary.qmd#tiered-storage). Its two **advisory** floors appear throughout the cases below: about 32 MB mean object size for the **warm tier** on disk, and about 100 MB for the **cold tier** on tape. Nothing enforces these floors. They keep the object count at a level that the shared index of the service can manage. It is not the only kind of object store, and its numbers are not a general rule. ::: {.callout-note} ## Cloud-optimized != smaller storage size @@ -29,10 +24,12 @@ The examples use public Copernicus and NASA missions. The [CNES Datalake](https: Cloud-optimized formats provide partial access via byte-range indices into the data, accessible via addressable metadata. **The extra bytes are the feature.** Overviews, levels of detail, and spatial indexes are structures the source file did not carry at all. They add volume, and they are exactly what buys fast display and cheap subsetting. The COG in case 1 is about 30% larger than the delivered archive mostly because it carries overviews, and the row-group index in case 4 is the whole point of the conversion. + +**Where the overviews live is the real trade-off.** Overview depth costs a COG only bytes, because its overviews sit inside the same file. It costs a Zarr store objects, because each pyramid level is a separate array with its own objects. Cases 1 and 2 measure this directly, and it is the most important result in this set. ::: ::: {.callout-note} -The numbers below are first measurements on a representative unit per mission, meaning one scene, one tile, or a few granules. They are illustrative, not a published benchmark. Two of the six cases have re-runs pending, and each one says so. +The numbers below come from the final benchmark campaign. Each figure is measured on a representative unit: an area block of 16 to 20 products for Sentinel-1 and Sentinel-2, and 2 to 12 granules or passes for the SWOT products. Every size comparison is like-for-like on content, and any content subset is flagged as one. The compression ratio, the codec contrast, and the tier classification are per-unit properties, so they carry to a full collection. The object count and the total volume scale with the number of units. They were produced with [`cng-formats-benchmark`](https://github.com/developmentseed/cng-formats-benchmark), an open-source harness that measures object-size distribution, write, read, and display per format. You can describe your own datasets in its config files and reproduce this shape of comparison on your own data. See [Optional: make it quantitative](index.qmd#optional-make-it-quantitative). ::: @@ -49,17 +46,25 @@ They were produced with [`cng-formats-benchmark`](https://github.com/development Both intents are served per scene, which is why this case does not need a cube at all. -**Measured, one scene.** Both layouts cleared the [warm tier](../glossary.qmd#tiered-storage). COG came to 2625.7 MiB in 28 objects. A scene-size GeoZarr shard came to 1975.6 MiB in 26 objects, about 25 % smaller. That gap is the **codec**, Deflate against ZSTD, not the format. Against the 2023.7 MiB delivered `.zip`, the GeoZarr store is about source size (−2 %) and the COG is larger (+30 %), because the COG carries overviews. +**Measured, 16-product area block.** Three arms carried the same pixels. COG came to 25478 MiB with ZSTD, in 448 objects at a mean object size of 57 MiB, which clears the [warm tier](../glossary.qmd#tiered-storage). A flat Zarr store with no overview pyramid came to 19432 MiB, about 24 % smaller. A pyramid-complete GeoZarr came to 26026 MiB, about 2 % *larger* than the COG, in 2510 objects at a mean of 10.4 MiB, which clears no tier at all. + +**The codec is not the driver. The pyramid is.** Swapping COG's own codec from ZSTD to Deflate changes its total by 2 %, from 25478 MiB to 25998 MiB. The 24 % that the flat Zarr store saves is the pyramid that it does not carry. Add that pyramid, and the saving disappears. -**[Interpretation encoding](index.qmd#interpretation-encoding) is a real differentiator here.** The source reflectance is stored as scaled integers, and the nodata fill is left undeclared. COG puts the scale, the offset, and the nodata in side metadata that readers honor unevenly. Zarr v3 can carry all three as first-class array properties, using the `scale_offset` codec and a declared fill value, so every reader recovers the same physical value. +**The pyramid buys display speed and costs tier fit.** With the pyramid, and with a reader that selects the matching multiscale level, zoomed-out display runs about 5x faster than the same store without a pyramid: 0.198 s against 0.98 s. The price is the object count, 5x more than that flat store, which takes this product below every tier. COG pays neither price, because its overviews sit inside the one file. + +**Against the delivered `.zip`, per scene.** The COG is 2625.7 MiB against a 2023.7 MiB source, so about 30 % larger. The overviews account for most of that. + +**[Interpretation encoding](index.qmd#interpretation-encoding) is a real differentiator here.** The source reflectance is stored as scaled integers, and the nodata fill is left undeclared. COG keeps the scale, the offset, and the nodata value in locations that the format does not standardize, such as a GDAL metadata tag. Readers do not apply them consistently. Zarr v3 can carry all three as first-class array properties, using the `scale_offset` codec and a declared fill value, so every reader recovers the same physical value. **Where it lands.** COG per scene for distribution and visualization. -For temporal analysis GeoZarr is the right target, but **the GeoZarr store measured here does not yet deliver it.** What was written is a scene-size store: one scene, so there is no time dimension to exploit. It proves the container and the encoding, because it comes out at roughly source size and it carries the scale, offset, and nodata correctly. A time-series query against it still has to open one store per date, which is the same fan-out the COG layout has. +For temporal analysis GeoZarr is the right target, but **the GeoZarr store measured here does not yet deliver it.** What was written is a per-scene, two-dimensional store, so there is no time dimension to exploit. It proves the container and the encoding, and it carries the scale, offset, and nodata correctly. A time-series query against it still has to open one store for each date, which is the same fan-out that the COG layout has. + +A pyramid-less store measures smaller, as the numbers above show. Do not read that as the efficient option. Such a store is valid GeoZarr, because multiscales are optional in the convention, but it cannot serve a web map at speed. The temporal benefit appears only once many scenes are concatenated along time into a single cube. That is the regime measured in [case 2](#case-time-cube), and it is a collection-level layout decision, not something a scene-size store can give you. Read the per-scene store as a building block and the cube as the product. -The associated masking product, which is written as many small GeoTIFFs, is a clear anti-pattern to fix at the source. +The related masking product is written as many small GeoTIFFs. Their mean size is far below the warm-tier floor. This is the small-object problem that the [grouping lever](index.qmd#grouping-lever) describes, and the producer must correct it at the source. Sentinel-2 also has a second multiscale dimension worth naming: its bands are natively captured at 10 m, 20 m, and 60 m, not derived from each other by downsampling. See [Multiscale and native mixed resolution](index.qmd#multiscale-and-native-mixed-resolution) for how one aligned pyramid can host all three without picking a single working resolution. @@ -75,7 +80,19 @@ Sentinel-2 also has a second multiscale dimension worth naming: its bands are na The analysis half is what separates this case from case 1: it is asked over time, not per scene. A year of RTC over a region is naturally a `time × y × x × pol` cube, and on a Zarr cube a time-series slice or an area-of-interest subset is one range-addressable query. Across thousands of per-timestamp COGs the same request becomes a large fan-out. Sharding reconciles fine-grained chunk access with the object store's preference for large objects. -**Measured, one tile and confirmed at area scale.** COG came to 728 MiB per tile and a scene-size GeoZarr shard to 545 MiB. Both cleared the [cold tier](../glossary.qmd#tiered-storage). GeoZarr is again about 25 % smaller, and again this is the codec: Deflate at 3.70x against ZSTD at 4.98x. The result held over a 16-tile block, at 16498 MiB for COG and 12344 MiB for GeoZarr. Against the 496 MiB delivered `.zip` both are larger, COG by 47 % and GeoZarr by 10 %. Cloud-optimized costs storage here, and buys the range access that the opaque zip cannot give at all. +**Measured, 20-product area block.** COG came to 16449 MiB with ZSTD, in 40 objects at a mean of 411 MiB, which clears the [cold tier](../glossary.qmd#tiered-storage). A flat Zarr store with no overview pyramid came to 12344 MiB in 80 objects, about 25 % smaller, and it also clears the cold tier. A pyramid-complete GeoZarr came to 15867 MiB, about 3.5 % *smaller* than the COG, but in 480 objects at a mean of 33 MiB, which drops it from the cold tier to the warm tier. + +**The codec is not the story here either.** Deflate against ZSTD changes the COG total by 0.3 %, from 16498 MiB to 16449 MiB. Sentinel-1 backscatter carries speckle, and speckle resists general-purpose compressors whatever the algorithm. + +**The pyramid buys display speed and costs tier fit.** With the pyramid, and with a reader that selects the matching multiscale level, zoomed-out display runs about 10x faster than the same store without a pyramid: 0.171 s against 1.82 s. The speed also holds across zoom levels instead of collapsing at low resolution. The price is 6x the object count of that flat store, and one tier level. + +**Against the delivered `.zip`, per scene.** The COG is 728 MiB against a 496 MiB source, so about 47 % larger. Cloud-optimized costs storage here, and it buys the range access that the opaque zip cannot give at all. + +::: {.callout-important} +**The cube itself is not measured.** Every GeoZarr figure above describes a per-scene, two-dimensional store. That is the shape in which Zarr has least to offer. A true time cube, one tile across many dates with the chunks laid over `time`, `y`, and `x`, is the shape that this recommendation is actually about, and this study did not test it. Nothing blocks the test technically. Read the case as an argument for the cube, and the numbers as evidence about the per-scene shape only. +::: + +**Where it lands.** COG for the serving path, because it keeps both the tier and the display speed. GeoZarr for the analysis path, once the cube is built and measured. ## 3. Point cloud: netCDF to COPC @@ -89,23 +106,29 @@ The analysis half is what separates this case from case 1: it is asked over time The fusion intent matters most here: these points get combined with other water products, so a reader has to recover the same physical value per point and subset the same area cheaply. That is what makes the source's missing spatial index a real cost rather than an inconvenience. COPC is a clustered octree inside a LAZ file; the netCDF source has no spatial index and no level of detail, so reading a small geographic subset means scanning the whole granule. -**Measured, content-complete, three granules.** A COPC carrying all per-point variables was about 16 % **smaller** than the source netCDF. This is the one clear size win among the cases, and it comes from the codec: LASzip at 3.72x against netCDF Deflate at 2.56x. Each granule cleared the tier on its own. +**Measured, content-complete, six granules.** The COPC came to 3518.6 MiB against 3621.4 MiB of source netCDF. That is size parity: about 3 % smaller in total, and between 6 % smaller and 5 % larger for each granule. LASzip does out-compress the source, at 2.72x to 3.45x against 2.56x, but not by enough to shrink the product. Each granule clears both tiers on its own, at a mean object size of 586 MiB. The real gain is what the netCDF lacks: - A **spatial index in three dimensions**. The octree subdivides in x, y *and* z, so a query can constrain a bounding box **and** an elevation range, reading only the overlapping nodes. That matters here because water surface elevation is the measurement, not merely a coordinate. -- A **level of detail**. Octree levels 0 to 11 are the point-cloud analogue of COG overviews. +- A **level of detail**. The octree holds 9,925 to 17,635 nodes for each granule, at a depth of 9 to 10. This is the point-cloud analogue of COG overviews. + +**The index has a price.** An octree-node spatial query costs 3.3 s to 3.8 s on these granules, and about 7 s at four-way concurrency. That is the cost of a tree deep enough to answer the query correctly. + +**Geometry precision decides whether the octree is usable.** PIXC stores longitude and latitude in degrees, and height in metres. An octree cube sized on those raw extents splits on height instead of on the footprint, and the single LAS scale that follows quantizes longitude and latitude to about a metre. The measured build avoids both problems: it reprojects to a local UTM zone before it builds the octree, it writes that CRS into the file, and it thins each coarse level into a real overview. ::: {.callout-caution} An earlier measurement on this product looked like a 97 % size saving. It had dropped most of the per-point variables. Size comparisons must be like for like on content. ::: -**Where it lands.** COPC. The cloud-optimized value is partial access plus level of detail, with a modest size win on top. GeoParquet is the alternative where columnar analytics dominate, though 3D pruning there is optional: the [spec](https://geoparquet.org/releases/v1.1.0/) allows `zmin` and `zmax` in the bbox covering, but the writer has to populate them and the engine has to use them. +**Where it lands.** COPC. The cloud-optimized value is partial access and level of detail at size parity, not a size saving. GeoParquet is the alternative where columnar analytics dominate, though 3D pruning there is optional: the [spec](https://geoparquet.org/releases/v1.1.0/) allows `zmin` and `zmax` in the bbox covering, but the writer has to populate them and the engine has to use them. ## 4. Vector and tabular: shapefile to GeoParquet **Product:** SWOT L2_HR Lake Single Pass. **Source format:** shapefile. +One pass is one overflight of the satellite. The mission delivers one shapefile for each pass. Each shapefile reports the whole prior lake database, and not only the lakes that the pass observed. About 78 % of the rows therefore describe a lake with no observation, and they carry a NULL geometry. The GeoParquet output keeps the same rows. + - **Existing data?** Yes. - **[Can you reprocess?](index.qmd#control)** Yes, so the format is a free choice. - **[What shape are the data?](index.qmd#topology)** Vector and tabular, which opens `{GeoParquet, FlatGeobuf}`. @@ -116,7 +139,7 @@ The intent mix makes this case a genuine two-way contest, though not in the way - **Download favors GeoParquet, not FlatGeobuf.** FlatGeobuf has [no internal compression](../flatgeobuf/intro.qmd#no-internal-compression), and it cannot simply be gzipped, because its spatial index addresses byte ranges in the *uncompressed* file. GeoParquet [compresses per column by default](../geoparquet/index.qmd#internal-compression-by-default), so within this candidate set it is the smaller transfer. - **Streaming is a visualization property, not a download one.** FlatGeobuf [streams](../flatgeobuf/intro.qmd#streaming-features-is-supported), so features render progressively as they arrive. Valuable, but visualization is absent from this mix. -- **FlatGeobuf wins on whole-row spatial subsetting.** It is row-oriented with a Hilbert R-tree, so every attribute of the features in a bounding box is one contiguous read, where a [column-oriented](../geoparquet/index.qmd#column-oriented) store has to touch every column. A real access pattern, just not one this intent mix asks for. +- **FlatGeobuf wins on whole-row spatial subsetting.** It is row-oriented with a Hilbert R-tree, so every attribute of the features in a bounding box is one contiguous read, where a [column-oriented](../geoparquet/index.qmd#column-oriented) store has to touch every column. The measurements below put this at about 10x faster. It is a real access pattern, but it is not one that this intent mix asks for. At volume the scale question would matter too, because GeoParquet shards across [multiple files](../geoparquet/index.qmd#multi-file-support) while FlatGeobuf is a single file with a single index. These passes are tiny, so it does not bite here. @@ -124,11 +147,17 @@ So GeoParquet wins on download *and* fusion. The product is fixed-schema tabular **Measured, two passes.** GeoParquet came to 17.8 MiB per pass, so 35.6 MiB for the two passes, against 31.1 MiB for the same two passes as a zipped shapefile. That is about 14 % **larger**, and the ratio is the same per pass (17.8 against 15.6 MiB). -Note that this compares GeoParquet against the **source**, not against FlatGeobuf. Against its actual rival in the candidate set, GeoParquet is the smaller of the two, because FlatGeobuf cannot compress at all. +Note that this compares GeoParquet against the **source**, not against FlatGeobuf. + +**Measured, three passes, both candidates.** GeoParquet with ZSTD came to 47.9 MB. FlatGeobuf with its R-tree index came to 75.1 MB, about 57 % larger. The R-tree itself is only about 2 % of the object, so the difference is the compression that FlatGeobuf cannot do. A bounding-box query, however, runs about 10x faster on FlatGeobuf: 0.049 s against 0.522 s warm. -Each pass sits below the warm-tier floor. The lever is therefore **aggregation across passes**: concatenate first, then size the row groups. A one-for-one transcode of each pass would leave the object-size problem in place. The gain over the shapefile is the bounding-box row-group index, not the size. +**FlatGeobuf could not hold the product as delivered.** A FlatGeobuf R-tree cannot index a NULL geometry, and 78 % of the rows carry one. The format could not take the product until those rows had a geometry. The run synthesized 20,689 geometries on the largest pass, from the reference point that each unobserved lake already carries in its attributes, and it recorded them as synthesized. This is a property of the shape of this product, not a fault in FlatGeobuf. It is also a good example of the data itself disqualifying a candidate. -**Where it lands.** GeoParquet, with pass-level aggregation. FlatGeobuf is the alternative where whole-row spatial subsetting or progressive rendering dominates, and where the data stays small enough for a single file. +**The codec question is closed.** Snappy is 12 % larger than ZSTD on this product, so ZSTD was already the better default. The 14 % against the zipped shapefile is a property of the format pair, not of a weak setting. + +Each pass sits below the warm-tier floor. The lever is therefore **aggregation across passes**: concatenate the rows from several passes into one file, then size the row groups. A one-for-one transcode of each pass would leave the object-size problem in place. The gain over the shapefile is the bounding-box row-group index, not the size. + +**Where it lands.** GeoParquet, with pass-level aggregation. It wins on size, on download, and on fusion. FlatGeobuf is the measured alternative where whole-row spatial subsetting dominates, and it is about 10x faster for that query, but it costs 57 % more storage and it needs the NULL geometries fixed first. ## 5. Small N-D grid: the aggregation regime @@ -140,11 +169,23 @@ Each pass sits below the warm-tier floor. The lever is therefore **aggregation a - **[Intent](index.qmd#intent).** Distribution and download, with data fusion against the other SWOT products, and some visualization. This mix is provisional and still to be confirmed with the product owner. - **[Grouping lever](index.qmd#grouping-lever).** Aggregation *above* the product, not sharding within it. A 100 m granule covers a small tile, about 160 km across, so it is far below the warm-tier floor. -**Measured, single variable, so not yet like for like.** On the primary water-surface-elevation variable, COG came to 4.3 MiB per granule and a GeoZarr shard to 0.8 MiB on average, because sharding fragments at this size. Both sit far below the warm-tier floor, so the lever is **aggregation**, and at this small size **COG is the cleaner carrier**: one tiled object that also holds overviews. An all-variable re-run is needed before quoting a size, for the same reason as case 3. +**Measured, 12 granules, all 38 CF variables.** One object for each variable gives 456 COG objects, at a mean of 2.55 MiB and 1107.8 MiB in total. The GeoZarr equivalent gives 2640 objects, at a mean of 471 KiB and 1214.8 MiB in total. Neither clears any tier. + +**The conversion creates a small-object problem that the source does not have.** The source is one netCDF for each granule. One object for each of its 38 variables multiplies the object count by 38, and the smallest COG object is 19 KB. The framework calls this the small-object anti-pattern, and here the conversion introduces it. -The natural aggregation unit is a spatial mosaic, per grid zone and per orbit cycle. That turns a wall of tiny granules into one range-addressable object. +**The GeoZarr side is worse for a configuration reason, not a format reason.** The shard was set to 2048 px against a grid of about 1457 px. The shard was therefore wider than the whole array, it clamped to a single chunk, and the sharding lever grouped nothing. One store for each variable also repeats the `x` and `y` coordinate arrays and the pyramid metadata 38 times. A smaller chunk shape was not tried. + +::: {.callout-caution} +**The content-subset trap, in numbers.** A COG of the water-surface-elevation variable alone is 4.3 MiB, against about 67 MiB for the whole granule. The apparent 92 % saving is dropped content, not compression. Compare like for like on content, as in case 3. +::: + +The natural aggregation unit is a spatial mosaic, for each grid zone and each orbit cycle. That turns a wall of tiny granules into one object that a reader can address with range requests. + +::: {.callout-warning} +**Aggregation is sized, not measured.** This study wrote no consolidated object and read none. The decisive question stays open: does a large aggregate stay range-addressable? A bounding-box read on the aggregate must cost about what the same read costs on the small units. Treat every aggregation figure in these cases as arithmetic, never as a measurement. +::: -**Where it lands.** COG for the small raster, plus collection-level aggregation. GeoZarr is the alternative to compare once the pyramid and all variables are in place. +**Where it lands.** COG for the small raster, plus aggregation across variables and granules. GeoZarr is the alternative to compare once the shard is sized against this grid. This case is the counterpart to cases 1 and 2. There, one large band already clears the tier, so you group *within* a product. Here the products are small, so you group *above* a single product. @@ -172,14 +213,14 @@ This case is also where the ["not yet possible"](index.qmd#not-yet-possible) fra | Case | Topology | Lands on | Size against source | Main gain | | --- | --- | --- | --- | --- | -| 1. Optical scene | 2D raster | COG per scene; a cube needs cross-scene aggregation | COG +30 %, GeoZarr −2 % | Partial access, encoding | -| 2. SAR time-cube | Raster over time | GeoZarr v3 sharded | COG +47 %, GeoZarr +10 % | One query instead of a fan-out | -| 3. Pixel cloud | Point cloud | COPC | −16 % | 3D spatial index, level of detail | +| 1. Optical scene | 2D raster | COG per scene | COG +30 % | Display speed at full tier fit | +| 2. SAR time-cube | Raster over time | COG to serve, GeoZarr to analyse | COG +47 % | One query instead of a fan-out | +| 3. Pixel cloud | Point cloud | COPC | parity, −3 % | 3D spatial index, level of detail | | 4. Lake vector | Vector, tabular | GeoParquet, aggregated | +14 % | Row-group bbox index | -| 5. Small raster grid | N-D grid | COG, plus aggregation | Re-run pending | Fewer, larger objects | +| 5. Small raster grid | N-D grid | COG, plus aggregation | not computed | Fewer, larger objects | | 6. Frozen archive | Chunked arrays only | Virtual Zarr | No rewrite | Range access with no reprocessing | -Read the size column together with the recurring lesson at the top of this page. Only one case got smaller, and that case got smaller because of its codec. +Read the size column together with the note at the top of this page. **No case came out smaller than its source.** Case 3 is the closest, and it lands at parity. In cases 1 and 2, the choice between the two candidates is not about size at all: both carry the same pixels for about the same bytes once both carry overviews. The choice is about where the overviews live, and what that costs in objects. ## Acknowledgements diff --git a/decision-framework/index.qmd b/decision-framework/index.qmd index e178cd68..e6529cc3 100644 --- a/decision-framework/index.qmd +++ b/decision-framework/index.qmd @@ -56,7 +56,7 @@ The framework does not restate that definition. It assumes it. Two consequences ## The two stages -The framework runs in **two stages**, and the topology question marks the hand-off between them. +The framework runs in **two stages**. The topology question is the point where Stage 1 ends and Stage 2 begins. **Stage 1 — control: do you have a choice?** Do the data already exist? If they exist, are you able (technically, legally) to reprocess them? This stage can end @@ -69,8 +69,8 @@ when Stage 1 leaves you free to choose. Here the shape of the data selects a shape. A *family*, not a format. Two or three candidates normally come through, and the refinements below rank them. -The hand-off is worth stating outright, because the diagram can be read as if -the shape of the data were the answer: +The change from Stage 1 to Stage 2 needs a clear statement. A reader can +interpret the diagram as if the shape of the data gives the answer: | Before the topology question | After it | | --- | --- | @@ -207,23 +207,29 @@ Most real datasets serve several intents at once. Weigh them in proportion rathe ### The object-size and grouping lever {#grouping-lever} -Every candidate format has a knob that sets how bytes are grouped into objects: +Two different controls are often both called the grouping lever. Keep them apart. One control sets the size of each object. The other control sets how finely a reader can address the bytes inside that object. -| Format | Grouping lever | -| --- | --- | -| COG | internal tiling and overviews | -| Zarr v3 | chunks within shards ([sharding](../glossary.qmd#chunk-size)) | -| COPC | octree nodes | -| GeoParquet | row groups, plus partitioning | +| Format | What sets the object size | What sets the read granularity inside the object | +| --- | --- | --- | +| COG | the extent of the file: area, bands, and time span | internal tiling and overviews | +| Zarr v3 | the shard size, because each shard is an object | the [chunk size](../glossary.qmd#chunk-size) inside the shard | +| COPC | the extent of the file: tile area and point density | octree nodes | +| GeoParquet | the partitioning: one file for each partition | row groups | + +Only Zarr connects the two controls. In a Zarr store, each chunk or shard is itself an object. In the other formats, one file is one object. Internal tiling and row groups do not change the object size. Tune the lever to your [object store's](../glossary.qmd#object-storage-cloud-storage) **object-size policy** (S3, Azure Blob Storage, Google Cloud Storage, or an on-premises equivalent). Object stores degrade when they must index very many tiny objects, so most reward a larger average object. -One worked example: the [CNES Datalake](https://cnes.fr/en/projects/centre-de-calcul) sorts data into [tiers](../glossary.qmd#tiered-storage) by mean object size, with about 32 MB or more for warm disk and about 100 MB or more for cold tape. This serves the general point: find out whether your storage has an object-size policy, and size the lever against it. Some providers publish a threshold, some only charge you as if your objects were larger, and some have no policy at all. +One worked example is the [CNES Datalake](https://cnes.fr/en/projects/centre-de-calcul), an in-house S3 service that moves data between disk and tape. Each [tier](../glossary.qmd#tiered-storage) recommends a minimum mean object size, measured as the total volume divided by the object count: about 32 MB for disk, and about 100 MB for tape. -The lever works in both directions, and which direction you need depends on product size: +These floors are advisory, and nothing refuses your write. They protect the shared object index, which holds one entry for each object. Too many small objects fill that index before the disk is full, and the Datalake then cannot supply its pooled volume to any project. -- **Products that are already large.** Group *within* one product. Size the shard, the tile, or the row group so it clears the policy. -- **Products that are small.** Sharding within one granule only makes the fragmentation worse. Group *above* the product instead: mosaic or concatenate many granules into one range-addressable object. +Your own storage service has its own policy, or it has none. Find the policy, then set the lever against it. Some providers publish a threshold. Other providers apply a minimum billable object size instead. [Amazon S3](https://docs.aws.amazon.com/AmazonS3/latest/userguide/storage-class-intro.html) charges a minimum of 128 KB for each object in Standard-IA, One Zone-IA, and Glacier Instant Retrieval. A 10 KB object then costs the same as a 128 KB object. The write does not fail, so you do not see this cost until the invoice. + +The correct direction depends on the size of your product: + +- **Products that are already large.** For Zarr, set the shard size so that the mean object size clears the policy. For a single-file format, the file is already one object. To make the objects smaller, split the file by region, by band, or by time step. +- **Products that are small.** Group *above* the product. Mosaic or concatenate many granules into one object that a reader can address with range requests. For Zarr, a larger shard does the same job. A finer internal layout inside one small granule makes the problem worse. **Size and shape are two different questions.** How *big* the objects are is a question for your object store, answered above. How the bytes are arranged *inside* them @@ -290,7 +296,7 @@ Does the format carry the information a reader needs to recover the same physica This axis separates the candidates more than people expect: - **Zarr v3 makes the information first-class.** With the `scale_offset` codec, a declared fill value, Zarr conventions for projection, CRS and multiscale, the transform is a first-class array property. Every reader recovers the same value. -- **COG, GeoParquet, and COPC standardize the CRS** but leave the value transform to side metadata. Readers honor side metadata unevenly. +- **COG, GeoParquet, and COPC standardize the CRS**, but they do not standardize the value transform. A COG that holds scaled integers usually keeps its scale and offset in a GDAL metadata tag, or in a STAC property, or only in the product handbook. These locations are conventions. They are not part of the data model of the format. One reader applies the transform, and another reader returns the raw integers. If your product stores scaled integers, or if an undeclared fill value would change a computed result, weigh this axis heavily. @@ -348,34 +354,19 @@ Frame the branch that way, and route the reader back to the options: reprocess i ## Optional: make it quantitative -The two stages above get you to a short list. When two or three candidates remain and the choice is close, a transparent score can make the trade-off explicit rather than a matter of taste. This section is **optional**. The framework stands on its own without it. - -The idea is a **weighted mean of criteria**, each expressed as a number from 0 to 1. It is framework-first and metrics-optional: qualitative terms alone already give a ranking, and a benchmark run sharpens the same ranking. +The two stages above give you a short list. If two or three candidates remain, and the choice is close, measure them. This section is **optional**. The framework works without it. -| Group | Criterion | -| --- | --- | -| Qualitative, no benchmark needed | **use-case fit**: your intent mix, applied to a per-format suitability matrix | -| | **maturity**: the L1 to L5 level above, mapped to a utility (`level / 5`) | -| | **interpretation encoding**: how much the format carries in-format, mapped to a utility | -| Measured, needs a benchmark run | **size**: output volume against the source | -| | **performance**: measured read, write, and display, blended by the intent mix | - -**Where the measured numbers come from.** The two measured criteria need a harness, and reproducibility is the whole point of quoting a number at all. [`cng-formats-benchmark`](https://github.com/developmentseed/cng-formats-benchmark) is the open-source (MIT) harness behind the numbers on the [Case Studies](case-studies.qmd) page. Per format and per dataset it measures the object-size distribution and tier fitness, write throughput, range-request-aware read latency, and tile-display latency. You describe your own datasets in config files, so no provider or dataset is baked into the code, and you run it on your own infrastructure against your own data. See the [documentation](https://developmentseed.github.io/cng-formats-benchmark/) and the [configuration guide](https://developmentseed.github.io/cng-formats-benchmark/configuration/). - -Rank the candidates and read the **margin**, which is the gap to the runner-up. A small margin is an honest signal that a benchmark, or a review of your weights, is what actually decides it. There are **no verdicts and no probability readout**. You get a score, a ranking, and a margin. +[`cng-formats-benchmark`](https://github.com/developmentseed/cng-formats-benchmark) is an open-source harness for this task. You describe your datasets in configuration files. The harness then measures the object-size distribution, the write throughput, the read latency, and the tile-display latency for each format. It also gives an optional weighted score, so you can rank the candidates against your own intent mix and read the margin to the second candidate. The numbers on the [Case Studies](case-studies.qmd) page come from this harness. -Two points are easy to get wrong: - -- **Object-size and tier fit is reported information, not a scored penalty.** Report whether a layout clears a given tier, the mean and median object size, and a small-file warning **beside** the score, never inside it. An object store's policy is a property of that store, not of the format, so you must weigh it against your own infrastructure. -- **Intent weights are your input.** The suitability matrix is a shared expert reference on each format's strengths, and it is useful on its own. The intent mix and the criterion weights belong to your context. - -The convincing artifact is never a single score. It is a sensitivity sweep that shows the ranking holds across a range of weights. That is also the honest reason this section stays short and links out: a score you cannot reproduce on your own data is not worth much, so the harness matters more than any number this page could print. +Two points are easy to get wrong if you score the candidates. Keep the object size and the tier fit beside the score, and not inside it, because a storage policy is a property of your infrastructure and not of the format. Set the weights yourself, because they belong to your context and not to the data. The [harness documentation](https://developmentseed.github.io/cng-formats-benchmark/) gives the details. ## Worked examples Six products run through the framework on the [Case Studies](case-studies.qmd) page. Five are backed by measurements. -One lesson recurs often enough to state up front: **going cloud-optimized is rarely smaller than the source you started from.** Overviews, levels of detail, and spatial indexes are structures the source did not carry, so they add volume, and that added volume is the feature you are buying. Where two candidates do differ in size, the cause is usually the **codec**, not the format. +One lesson recurs often enough to state up front: **going cloud-optimized is rarely smaller than the source you started from.** Overviews, levels of detail, and spatial indexes are structures the source did not carry, so they add volume, and that added volume is the feature you are buying. + +Where two candidates do differ in size, look at the overviews before you look at the codec. In the measured cases, a codec change moves the total by 0.3 % to 2 %. Adding an overview pyramid to a store that had none adds 28 % to 34 %. The pyramid also costs a COG only bytes, because its overviews sit inside the one file, and it costs a Zarr store objects, because each level is a separate array. ## Acknowledgements diff --git a/glossary.qmd b/glossary.qmd index 89aeafec..3885996a 100644 --- a/glossary.qmd +++ b/glossary.qmd @@ -338,7 +338,7 @@ A file format for image and raster data that supports [lossless compression](#lo A storage service that keeps data on different classes of hardware at different prices and speeds. A common arrangement is a "warm" tier on disk for data that is read often, and a "cold" tier on tape or archival storage for data that is read rarely. -Some providers set a minimum mean [object](#object-storage-cloud-storage) size for each tier, because indexing very many small objects is expensive. A layout whose objects fall below that floor can be refused or moved to a slower tier, so [chunk size](#chunk-size) and grouping choices have to be made against the policy of the storage you are actually using. +Some providers set a recommended minimum mean [object](#object-storage-cloud-storage) size for each tier. The storage service holds one index entry for each object, and very many small objects can fill that index before the storage capacity is full. Some providers enforce this limit. Other providers give it as guidance only. Find the policy of the service that you use, then make your [chunk size](#chunk-size) and grouping choices against it. #### Vector data From 22dec0b2418bf5e6502fdc50b7bc79baa7431d28 Mon Sep 17 00:00:00 2001 From: Aimee Barciauskas Date: Wed, 9 Sep 2026 07:58:23 -0700 Subject: [PATCH 56/59] More changes --- decision-framework/index.qmd | 30 +++++++++++------------------- 1 file changed, 11 insertions(+), 19 deletions(-) diff --git a/decision-framework/index.qmd b/decision-framework/index.qmd index 008e9634..3b1ac255 100644 --- a/decision-framework/index.qmd +++ b/decision-framework/index.qmd @@ -7,7 +7,7 @@ subtitle: "How to go from data you must publish to a short list of cloud-optimiz This framework helps you go from *"I have geospatial data to publish or migrate"* to a **short list of cloud-optimized format candidates**, and to the **layout choices** that make each candidate work. -It is a guide, not an oracle: +It is a guide to help make informed decisions, and does not prescribe a single correct format. - **What you control** comes first: whether the data already exist, whether you may reprocess them, and whether a mandate fixes an archival format. This can settle the answer before format even comes up. - The **shape of your data** narrows the set of candidate formats. @@ -48,7 +48,7 @@ The framework uses a few terms in a specific way. Each links to the section that This guide already defines what makes a format cloud-optimized. The Welcome page states it as a four-part [cloud-optimized format pattern](../index.qmd#the-opportunity), and the glossary defines [cloud-optimized](../glossary.qmd#cloud-optimized) as the ability to read a meaningful *part* of a file without downloading *all* of it. -The framework starts from that definition Two consequences are worth naming here, because they decide whether a candidate belongs in the framework at all: +The framework starts from the definition on the Welcome page. Two requirements are worth explaining in more detail, because they decide whether a candidate belongs in the framework at all: **P1. The format must support partial and parallel reads over HTTP.** [Random access](../glossary.qmd#random-access) through [HTTP range requests](../glossary.qmd#http-range-request) is what makes a format usable from [object storage](../glossary.qmd#object-storage-cloud-storage). It depends on the rest of the pattern: [metadata](../glossary.qmd#metadata) in a consistent place that a reader can load in a single read, and then addressable [chunks](../glossary.qmd#chunk), internal tiling, or both. A reader that cannot locate the bytes it needs from the metadata alone cannot fetch only those bytes. @@ -56,7 +56,7 @@ The framework starts from that definition Two consequences are worth naming here ## The two stages -The framework runs in **two stages**, and the topology question marks the hand-off between them. +The framework has **two stages**: **Stage 1 — control: do you have a choice?** Do the data already exist? If they exist, are you able (technically, legally) to reprocess them? This stage can end @@ -68,15 +68,6 @@ when Stage 1 leaves you free to choose. Here the shape of the data selects a **format family**: the small group of cloud-optimized formats built for that shape. The refinements below rank the options within a format family. -The hand-off is worth stating outright, because the diagram can be read as if -the shape of the data were the answer: - -| Before the topology question | After it | -| --- | --- | -| Do I have a cloud-optimized target at all? | Which of these targets suits me best? | -| The outcome can be "not yet possible" | Every remaining candidate works; you are ranking | -| Decided by ownership and reprocessing rights | Decided by intent, layout, maturity, encoding | - Four refinements then rank the candidates: **intent**, the **object-size and grouping lever**, **ecosystem maturity**, and **interpretation encoding**. The last one deserves a close look, because the formats differ on it more than @@ -129,7 +120,9 @@ flowchart TD **No existing data.** You are designing a new product, so you can choose freely. Go to the [topology question](#topology). -**If an archival-format mandate applies,** it adds a second job, and it does not remove your freedom to choose how readers get access. Applying cloud-optimized practice to the archive itself is not optional: whatever the mandated format, make it as range-request-friendly as it can be. What is genuinely optional is what you do beyond that. The stronger, recommended option is to also publish a full cloud-optimized copy for readers, since a native copy gives real range access rather than the archive's own chunking. A lighter option is to skip the second copy and instead wrap the optimized archive in a [virtual Zarr](../zarr/virtual-zarr.qmd) layer: cheaper to maintain, but you inherit the archive's chunk shape, the same limit as [existing data you cannot reprocess](#control) below. Where the mandated format is HDF5 or netCDF, this guide has [specific guidance](../cloud-optimized-netcdf4-hdf5/index.qmd) on consolidated metadata and chunk sizing. For other archival formats the principle holds even where the guide has no page: optimize what you must keep, then choose deliberately how far you go for readers. +**If an archival-format mandate applies,** it adds a second job, and it does not remove your freedom to choose how readers get access. Whatever the mandated format, make it as range-request-friendly as it can be. Where the mandated format is HDF5 or netCDF, this guide has [specific guidance](../cloud-optimized-netcdf4-hdf5/index.qmd) on consolidated metadata and chunk sizing. What is genuinely optional is what you do beyond the archival format. + +You can wrap the optimized archive in a [virtual Zarr](../zarr/virtual-zarr.qmd) layer: cheaper to maintain, but you inherit the archive's chunk shape, the same limit as [existing data you cannot reprocess](#control) below. The stronger, recommended option is to also publish a cloud-optimized copy for readers, since a cloud-optimized copy is an opportunity to provide access beyond the archive's native chunking. **Existing data you can reprocess.** You control the production chain, so you can also choose freely. Go to the [topology question](#topology). @@ -176,7 +169,7 @@ technically and legally free to change format and still be held by what its user tools expect. See [Existing users and their tools](#existing-users). -### Topology: what shape are the data? {#topology} +## Topology: what shape are the data? {#topology} The shape of the data selects a **format family**: the candidate set you refine within. It does not select a format. [Intent](#intent) is what decides between the members. @@ -192,7 +185,7 @@ The question is about topology, not about the format the data arrives in. A netC ## Refinements {#refinements} -Apply these within the candidate set your topology opened. +These refinements help rank candidate formats within a family. ### Intent {#intent} @@ -217,12 +210,11 @@ Every candidate format has a knob that sets how bytes are grouped into data chun Tune the lever to your [object store's](../glossary.qmd#object-storage-cloud-storage) **object-size policy** (S3, Azure Blob Storage, Google Cloud Storage, or an on-premises equivalent). Object stores degrade when they must index very many tiny objects, so most reward a larger average object. -One worked example: the [CNES Datalake](https://cnes.fr/en/projects/centre-de-calcul) sorts data into [tiers](../glossary.qmd#tiered-storage) by mean object size, with about 32 MB or more for warm disk and about 100 MB or more for cold tape. This serves the general point: find out whether your storage has an object-size policy, and adjust your groupings accordingly. Some providers publish a threshold, some only charge you as if your objects were larger, and some have no policy at all. +One worked example: the [CNES Datalake](https://cnes.fr/en/projects/centre-de-calcul) sorts data into [tiers](../glossary.qmd#tiered-storage) by mean object size, with about 32 MB or more for warm disk and about 100 MB or more for cold tape. This serves the general point: find out whether your storage has an object-size policy, and adjust your groupings accordingly. -The lever works in both directions, and which direction you need depends on product size: +Some providers publish a threshold, some only charge you as if your objects were larger, and some have no policy at all. -- **Products that are already large.** Group *within* one product. Size the shard, the tile, or the row group so it clears the policy. -- **Products that are small.** Sharding within one already small file only makes the fragmentation worse. Group *above* the product instead: mosaic or concatenate many small files into one range-addressable object. +Objects can be too large (readers have to download and decompress more data than they need) or too small (readers make many small requests and pay the network latency for each). **Size and shape are two different questions.** How *big* the objects are is a question for your object store, answered above. How the bytes are arranged *inside* them From a757b2fbc4492a32fcd999809c825a27164674ab Mon Sep 17 00:00:00 2001 From: Aimee Barciauskas Date: Wed, 9 Sep 2026 13:34:06 -0700 Subject: [PATCH 57/59] Add more wording changes --- decision-framework/index.qmd | 65 +++++++++++++----------------------- glossary.qmd | 4 +++ 2 files changed, 28 insertions(+), 41 deletions(-) diff --git a/decision-framework/index.qmd b/decision-framework/index.qmd index 3b1ac255..f70feab0 100644 --- a/decision-framework/index.qmd +++ b/decision-framework/index.qmd @@ -35,7 +35,7 @@ The framework uses a few terms in a specific way. Each links to the section that | [**Access pattern**](../glossary.qmd#access-pattern) | What subsets readers will request: a time series at one point, a spatial window at one date, a full scan. Intent implies an access pattern, and your layout either serves it or fights it. Defined in the guide's glossary; the layout consequences are in [the grouping lever](#grouping-lever). | | [**Archival-format mandate**](#control) | A requirement to maintain a specific archival format for preservation, such as HDF5, netCDF or GRIB, whatever cloud-optimized format you publish alongside it. | | [**Control**](#control) | Whether the choice is yours to make at all: do the data already exist, and may you reprocess them, technically and legally. This is Stage 1 of the framework, and it can end in "not yet possible". | -| [**Ecosystem maturity**](#maturity) | How settled a format is, on an L1 to L5 scale from experimental to de-facto standard. It answers "safe default, or bet?". | +| [**Ecosystem maturity**](#maturity) | How settled a format is, on an L1 to L5 scale from experimental to de-facto standard (see below). It answers "safe default, or bet?". | | [**Grouping lever**](#grouping-lever) | The knobs each format gives you for grouping bytes into data chunks and objects in cloud storage: COG internal tiling, Zarr v3 chunking and sharding, COPC octree nodes, GeoParquet row groups. | | [**Intent**](#intent) | What the data are for: archive and download, processing, visualization, distribution, or data fusion. Most products serve several intentsat once. | | [**Interpretation encoding**](#interpretation-encoding) | Whether the format carries units, CRS, nodata and any scale and offset transform in-format, so every reader recovers the same physical value, or leans on a side catalog. | @@ -116,7 +116,7 @@ flowchart TD ``` -### The control fork {#control} +## The control fork {#control} **No existing data.** You are designing a new product, so you can choose freely. Go to the [topology question](#topology). @@ -132,7 +132,7 @@ You can wrap the optimized archive in a [virtual Zarr](../zarr/virtual-zarr.qmd) **Neither.** Some sources cannot be repackaged or virtualized today. Examples are HDF4, encrypted files, and a single large blob with no internal structure, whether or not it's compressed. See [Not yet possible](#not-yet-possible). -#### How to tell which branch you are on +### How to tell which branch you are on Whether you can reprocess or virtualize usually depends on the data and data project context. @@ -171,7 +171,7 @@ tools expect. See [Existing users and their tools](#existing-users). ## Topology: what shape are the data? {#topology} -The shape of the data selects a **format family**: the candidate set you refine within. It does not select a format. [Intent](#intent) is what decides between the members. +The shape of the data selects a **format family**: the set of format candidates the refinement process narrows down. [Intent](#intent) is what decides between the members. | Data shape | Format family | Where each member wins | | --- | --- | --- | @@ -195,7 +195,7 @@ What are the data for? Archive and download, processing, visualization, distribu - **On-the-fly and batched processing pull the layout in opposite directions.** On-the-fly access rewards small chunks, fine overviews, and cache locality: a reader fetches a little and wants it back fast. Batched processing rewards large shards or row groups, so that a bulk scan becomes a few large contiguous reads that many workers can run in parallel, instead of thousands of small scattered ones. - **Data fusion raises the stakes on encoding.** Combining products from different missions or processors means every input must be read back as the same physical value, and subsets from each must line up. Fusion is a form of processing, but it is worth naming separately: it weights [interpretation encoding](#interpretation-encoding) and consistent grids far higher than single-product analysis does, and it usually rewards a tabular or columnar target where joins are cheap. -Most real datasets serve several intents at once. Weigh them in proportion rather than picking one. The [case studies](case-studies.qmd) state an intent mix for each product, which is what decides the ranking inside a candidate set. +Most real datasets serve several intents at once. However, the intents often carry different weight in the decision process. The [case studies](case-studies.qmd) state an intent mix for each product, which is what decides the ranking inside a candidate set. ### The object size and grouping lever {#grouping-lever} @@ -204,28 +204,28 @@ Every candidate format has a knob that sets how bytes are grouped into data chun | Format | Grouping lever | | --- | --- | | COG | internal tiling and overviews | -| Zarr v3 | chunks within shards ([sharding](../glossary.qmd#chunk-size)) | +| Zarr v3 | chunks and shards ([chunk](../glossary.qmd#chunk) and [shard](../glossary.qmd#shard)) | | COPC | octree nodes | | GeoParquet | row groups, plus partitioning | -Tune the lever to your [object store's](../glossary.qmd#object-storage-cloud-storage) **object-size policy** (S3, Azure Blob Storage, Google Cloud Storage, or an on-premises equivalent). Object stores degrade when they must index very many tiny objects, so most reward a larger average object. +Tune the lever to your [object store's](../glossary.qmd#object-storage-cloud-storage) **object-size policy** (S3, Azure Blob Storage, Google Cloud Storage, or an on-premises equivalent). Object stores degrade when they must index very many tiny objects, so most reward a larger average object. -One worked example: the [CNES Datalake](https://cnes.fr/en/projects/centre-de-calcul) sorts data into [tiers](../glossary.qmd#tiered-storage) by mean object size, with about 32 MB or more for warm disk and about 100 MB or more for cold tape. This serves the general point: find out whether your storage has an object-size policy, and adjust your groupings accordingly. +In general, when it comes to reading from object storage, objects can be too large (readers have to download and decompress more data than they need) or too small (readers make many small requests and pay the network latency for each). See also: [the datacube guid on tiny chunks](https://developmentseed.org/datacube-guide/latest/worst-practices/tiny-chunks.html) and [massive chunks](https://developmentseed.org/datacube-guide/latest/worst-practices/massive-chunks.html). -Some providers publish a threshold, some only charge you as if your objects were larger, and some have no policy at all. +One example: the [CNES Datalake](https://cnes.fr/en/projects/centre-de-calcul) sorts data into [tiers](../glossary.qmd#tiered-storage) by mean object size, with about 32 MB or more for warm disk and about 100 MB or more for cold tape. This serves the general point: find out whether your storage has an object-size policy, and adjust your groupings accordingly. -Objects can be too large (readers have to download and decompress more data than they need) or too small (readers make many small requests and pay the network latency for each). +Some providers publish a threshold, some only charge you as if your objects were larger, and some have no policy at all. **Size and shape are two different questions.** How *big* the objects are is a question for your object store, answered above. How the bytes are arranged *inside* them (chunk shape, and the order of the dimensions) is an **access-pattern** question, -and the object store has no opinion on it. A `time × y × x` cube chunked long in time +and an object store provider has no opinion about the shape. A `time × y × x` cube chunked long in time serves a time series at one point cheaply and a full-scene map expensively; chunked wide in `y` and `x` it does the opposite. This guide works it through, with a diagram, in [chunk shape vs chunk size](../cloud-optimized-netcdf4-hdf5/index.qmd#chunk-shape-vs-chunk-size). Shape rarely decides *which* format you pick, because every candidate in a family lets -you set it: a COG chooses its tile dimensions, Zarr its chunk shape and dimension +you set the shape: a COG chooses its tile dimensions, Zarr its chunk shape and dimension order, GeoParquet its row-group size and row ordering. What shape decides is how well the format you picked performs, not which one you pick. And it is mostly settled back at the [control](#control) question, because only reprocessing lets you choose it freely. @@ -239,7 +239,7 @@ To keep that judgement explicit rather than impressionistic, this framework assi | Level | Meaning | | --- | --- | -| **L5** | De-facto standard, universal tooling, no specification risk | +| **L5** | De-facto standard: universal tooling, no specification risk | | **L4** | Established: released spec, broad independent tooling, in production use | | **L3** | Stable: released spec, multiple implementations, growing tooling | | **L2** | Emerging: spec stabilizing, reference implementations, early adopters | @@ -267,11 +267,11 @@ This distinction is easy to miss, and it decides how much risk you actually take - **Zarr v3, the substrate of GeoZarr, is dependable.** The core specification was finalized in 2024. [Sharding](#grouping-lever), the feature that matters most for object storage, is stable in `zarr-python` 3.x and readable by GDAL and xarray. Judged on its own, the Zarr v3 core behaves like an L4. - **GeoZarr, the convention layer, is still being ratified.** It is what makes a generic [Zarr](../glossary.qmd#zarr) store *geospatial*: CRS, dimension semantics, and multiscale pyramids. It does for Zarr roughly what the GeoTIFF tags do for a plain TIFF. It is a draft in an OGC Standards Working Group, on the standards track but not yet an adopted OGC standard, and the published draft can lag behind active convention work. Check [geozarr.org](https://geozarr.org/) for the current status before you rely on a specific version. -So the L2 above belongs to the convention layer, not to Zarr. The leading reference implementations of GeoZarrtoday, such as ESA's EOPF Sentinel output, are demonstrators rather than production guarantees. +So the L2 above belongs to the convention layer, not to Zarr. The leading reference implementations of GeoZarr today, such as ESA's EOPF Sentinel output, are demonstrators rather than production guarantees. #### Maturity is a warning, not a veto -An L2 candidate can still be the right answer. Multidimensional cubes have no mature candidate for the multidimensional topology. A low level does not mean "choose something else". It means **plan for change**. Budget for convention churn, pin the versions you depend on, and expect visualization tooling in particular to improve as tooling catches up. +An L2 candidate can still be the right answer. Multidimensional cubes have no mature format candidate. A low level does not mean "choose something else". It means **plan for change**. Budget for convention churn, pin the versions you depend on, and expect visualization tooling in particular to improve as tooling catches up. Many factors influence what maturity level you can tolerate. A one-off internal product and a mission archive with a twenty-year retention commitment can reasonably land on different levels from the same set of candidate formats. @@ -304,15 +304,17 @@ Some choices sit below the format and are foundational: - Data types, and scale-offset encoding - Sharding in Zarr v3 -These are mature and in scope. The per-format pages cover them. More experimental optimizations, such as bit-rounding and advanced compression tricks, are out of scope here, or should be marked as immature when you use them. +These are mature and in scope. The per-format pages cover them. More experimental optimizations, such as bit-rounding and advanced compression tricks, are out of scope at this time. Formats also differ in which data types they support, and in whether types can be mixed within one file, column, or schema. Those specifics live on each format's own page and are worth checking before committing to a candidate; they can rule one out as surely as topology or intent. ### Multiscale and native mixed resolution -A multiscale pyramid does not have to come from one source resolution downsampled by a fixed factor. The common case is **overviews**, also called [pyramids](../glossary.qmd#overviews): downsampled copies of the same band, generated at a uniform decimation so a reader can zoom out without reading and resampling full-resolution chunks. COG overviews, COPC's octree levels of detail, and most GeoZarr pyramids work this way, and it is already covered above under [ecosystem maturity](#maturity) and the [grouping lever](#grouping-lever). +Providing downsampled levels of original data in the source resolution is a common pattern for providing good performance, usually for visualization. A common example is COG overviews, which provide tiles at standard resolutions for mapping clients. A multiscale pyramid is a more general concept: a set of levels of detail, each with its own resolution, that can be used for visualization or analysis. + +Often these multiscale pyramids are generated at a uniform decimation so a reader can zoom out without reading and resampling full-resolution chunks. COG overviews, COPC's octree levels of detail, and most GeoZarr pyramids work this way, and it is already covered above under [ecosystem maturity](#maturity) and the [grouping lever](#grouping-lever). -The same structure can also host bands that were never derived from each other and were simply captured at different native resolutions, as long as the steps between levels are chosen to land on each native resolution rather than a fixed factor. Sentinel-2 is the reference case, worked through in [case 1](case-studies.qmd) of the case studies: the 10 m, 20 m, and 60 m bands from the same sensor fold into one pyramid by aligning the decimation to each step (10 m to 20 m halves the resolution, 20 m to 60 m divides it by three), and the pyramid can continue past 60 m into ordinary overviews using the same mechanism. The [Zarr multiscales convention](https://github.com/zarr-conventions/multiscales) is built for exactly this: scale factors between levels are not required to be uniform, and its own reference examples use this Sentinel-2 layout. +How these multiscale pyramids are generated can vary: Sometimes, data is provided at varying resolution where lower resolution data is not derived from the higher resolution data. Sentinel-2 is an example, worked through in [case 1](case-studies.qmd) of the case studies: Sentinel-2 collects data in 10m, 20m, and 60m bands from the same sensor. These bands can become a pyramid by aligning the decimation to each step (10m to 20m halves the resolution, 20m to 60m divides it by three), and the pyramid can continue past 60 m into ordinary overviews using the same mechanism. [Zarr multiscales convention](https://github.com/zarr-conventions/multiscales) allows for non-uniform scale factors. This still interacts with the [grouping lever](#grouping-lever), since each native resolution needs its own tiling or chunk shape, and with [interpretation encoding](#interpretation-encoding), since each band needs its own geotransform. What changes from the plain-overview case is that these become refinements within one multiscale structure. @@ -334,34 +336,15 @@ In the short term the answer is usually the same: publish the cloud-optimized fo When a source cannot be repackaged or virtualized today, the honest answer is *"not yet implemented"*, not *"impossible"*. The blocker is usually a missing parser, an unsupported codec, or source files with no internal chunking at all. Every one of those can change. -The precedent is HDF5 itself: once the standard example of a format that could not be read efficiently from object storage, until a change in write strategy and reader tooling made it viable, see [Cloud-Optimized HDF/NetCDF](../cloud-optimized-netcdf4-hdf5/index.qmd).The reader half is still moving: switching xarray's default cache for the h5netcdf engine improves dataset opening ([xarray #11216](https://github.com/pydata/xarray/pull/11216), measured by its authors on NISAR data). Formats move, and so do the libraries. +The precedent is HDF5 itself: once the standard example of a format that could not be read efficiently from object storage, until a change in write strategy and reader tooling made it viable, see [Cloud-Optimized HDF/NetCDF](../cloud-optimized-netcdf4-hdf5/index.qmd). The reader half is still moving: switching xarray's default cache for the h5netcdf engine improves dataset opening ([xarray #11216](https://github.com/pydata/xarray/pull/11216), measured by its authors on NISAR data). Formats move, and so do the libraries. -Frame the branch that way, and route the reader back to the options: reprocess if you can get the chain changed, virtualize if a reader exists, or convert on ingest. Say what is missing, so somebody can unlock it. +Not yet possible does not mean never possible. Re-evaluate the options to pinpoint what needs to change to make cloud-optimized access possible. This could mean revaluating reprocessing, conversion and virtualization. ## Optional: make it quantitative -The two stages above get you to a short list. When two or three candidates remain and the choice is close, a transparent score can make the trade-off explicit rather than a matter of taste. This section is **optional**. The framework stands on its own without it. - -The idea is a **weighted mean of criteria**, each expressed as a number from 0 to 1. It is framework-first and metrics-optional: qualitative terms alone already give a ranking, and a benchmark run sharpens the same ranking. - -| Group | Criterion | -| --- | --- | -| Qualitative, no benchmark needed | **use-case fit**: your intent mix, applied to a per-format suitability matrix | -| | **maturity**: the L1 to L5 level above, mapped to a utility (`level / 5`) | -| | **interpretation encoding**: how much the format carries in-format, mapped to a utility | -| Measured, needs a benchmark run | **size**: output volume against the source | -| | **performance**: measured read, write, and display, blended by the intent mix | - -**Where the measured numbers come from.** The two measured criteria need a benchmark, and reproducibility is the whole point of quoting a number at all. [`cng-formats-benchmark`](https://github.com/developmentseed/cng-formats-benchmark) is the open-source (MIT) harness behind the numbers on the [Case Studies](case-studies.qmd) page. Per format and per dataset it measures the object-size distribution and tier fitness, write throughput, range-request-aware read latency, and tile-display latency. You describe your own datasets in config files, so no provider or dataset is baked into the code, and you run it on your own infrastructure against your own data. See the [documentation](https://developmentseed.github.io/cng-formats-benchmark/) and the [configuration guide](https://developmentseed.github.io/cng-formats-benchmark/configuration/). - -You can use this framework to rank the candidates and evaluate the **margin** between the winner and the runner-up. A small margin is an honest signal that a benchmark, or a review of your weights, is what actually decides it. There are **no verdicts and no probability readout**. You get a score, a ranking, and a margin. - -Two points are easy to get wrong: - -- **Object-size and tier fit is reported information, not a scored penalty.** Report whether a layout clears a given tier, the mean and median object size, and a small-file warning **beside** the score, never inside it. An object store's policy is a property of that store, not of the format, so you must weigh it against your own infrastructure. -- **Intent weights are your input.** The suitability matrix is a shared expert reference on each format's strengths, and it is useful on its own. The intent mix and the criterion weights belong to your context. +The two stages above get you to a short list. When two or three candidates remain and the choice is close, a transparent score can make the trade-off explicit and quantitative, reducing bias. -The convincing artifact is never a single score. It is a sensitivity sweep that shows the ranking holds across a range of weights. That is also the honest reason this section stays short and links out: a score you cannot reproduce on your own data is not worth much, so the harness matters more than any number this page could print. +[`cng-formats-benchmark`](https://github.com/developmentseed/cng-formats-benchmark) is an open-source (MIT) harness behind the numbers on the [Case Studies](case-studies.qmd) page. Per format and per dataset it measures the object-size distribution and tier fitness, write throughput, range-request-aware read latency, and tile-display latency. You describe your own datasets in config files, so no provider or dataset is baked into the code, and you run it on your own infrastructure against your own data. See the [documentation](https://developmentseed.github.io/cng-formats-benchmark/) and the [configuration guide](https://developmentseed.github.io/cng-formats-benchmark/configuration/). ## Worked examples diff --git a/glossary.qmd b/glossary.qmd index 89aeafec..0d7b9d36 100644 --- a/glossary.qmd +++ b/glossary.qmd @@ -318,6 +318,10 @@ An R library for using and managing [vector data](#vector-data), organized aroun A [vector data](#vector-data) file format. There are [many reasons](http://switchfromshapefile.org/) to no longer use Shapefile. +#### Shard + +The definition of shard here is specific to [Zarr](#zarr). A shard is a single object in object storage that contains multiple [chunks](#chunk) of data. See also: [Zarr sharding](https://zarr.readthedocs.io/en/latest/user-guide/performance/#sharding). + #### Space-filling curve An algorithm that translates two- or n- dimensional data into one-dimensional data. In practice, this is used as part of [spatial indexes](#spatial-index) to group [vector geometries](#vector-data) nearby within a file according to their two-dimensional location. From e287e2c3d8b4c5ffc52d768c2e5a7f603f5a8da6 Mon Sep 17 00:00:00 2001 From: "Alex I. Mandel" Date: Fri, 25 Sep 2026 09:40:39 -0700 Subject: [PATCH 58/59] typo Co-authored-by: Emmanuel Mathot --- decision-framework/index.qmd | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/decision-framework/index.qmd b/decision-framework/index.qmd index c00da605..ae77ad2a 100644 --- a/decision-framework/index.qmd +++ b/decision-framework/index.qmd @@ -37,7 +37,7 @@ The framework uses a few terms in a specific way. Each links to the section that | [**Control**](#control) | Whether the choice is yours to make at all: do the data already exist, and may you reprocess them, technically and legally. This is Stage 1 of the framework, and it can end in "not yet possible". | | [**Ecosystem maturity**](#maturity) | How settled a format is, on an L1 to L5 scale from experimental to de-facto standard (see below). It answers "safe default, or bet?". | | [**Grouping lever**](#grouping-lever) | The knobs each format gives you for grouping bytes into data chunks and objects in cloud storage: COG internal tiling, Zarr v3 chunking and sharding, COPC octree nodes, GeoParquet row groups. | -| [**Intent**](#intent) | What the data are for: archive and download, processing, visualization, distribution, or data fusion. Most products serve several intentsat once. | +| [**Intent**](#intent) | What the data are for: archive and download, processing, visualization, distribution, or data fusion. Most products serve several intents at once. | | [**Interpretation encoding**](#interpretation-encoding) | Whether the format carries units, CRS, nodata and any scale and offset transform in-format, so every reader recovers the same physical value, or leans on a side catalog. | | [**Object size**](#grouping-lever) | The size of the objects your layout actually writes to storage. Object stores reward a larger average object and many apply a minimum size policy. | | [**Refinement**](#refinements) | A criterion applied *inside* a format family to rank its members. There are four: intent, the grouping lever, ecosystem maturity, and interpretation encoding. | From 5b208537c4863bf6cd59a121fbdb957588afd338 Mon Sep 17 00:00:00 2001 From: "Alex I. Mandel" Date: Fri, 25 Sep 2026 09:40:55 -0700 Subject: [PATCH 59/59] typo Co-authored-by: Emmanuel Mathot --- decision-framework/index.qmd | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/decision-framework/index.qmd b/decision-framework/index.qmd index ae77ad2a..8092aff9 100644 --- a/decision-framework/index.qmd +++ b/decision-framework/index.qmd @@ -210,7 +210,7 @@ Every candidate format has a knob that sets how bytes are grouped into data chun Tune the lever to your [object store's](../glossary.qmd#object-storage-cloud-storage) **object-size policy** (S3, Azure Blob Storage, Google Cloud Storage, or an on-premises equivalent). Object stores degrade when they must index very many tiny objects, so most reward a larger average object. -In general, when it comes to reading from object storage, objects can be too large (readers have to download and decompress more data than they need) or too small (readers make many small requests and pay the network latency for each). See also: [the datacube guid on tiny chunks](https://developmentseed.org/datacube-guide/latest/worst-practices/tiny-chunks.html) and [massive chunks](https://developmentseed.org/datacube-guide/latest/worst-practices/massive-chunks.html). +In general, when it comes to reading from object storage, objects can be too large (readers have to download and decompress more data than they need) or too small (readers make many small requests and pay the network latency for each). See also: [the datacube guide on tiny chunks](https://developmentseed.org/datacube-guide/latest/worst-practices/tiny-chunks.html) and [massive chunks](https://developmentseed.org/datacube-guide/latest/worst-practices/massive-chunks.html). One example: the [CNES Datalake](https://cnes.fr/en/projects/centre-de-calcul) sorts data into [tiers](../glossary.qmd#tiered-storage) by mean object size, with about 32 MB or more for warm disk and about 100 MB or more for cold tape. This serves the general point: find out whether your storage has an object-size policy, and adjust your groupings accordingly.