Add format-selection decision framework page and case studies

.gitignore

@@ -15,3 +15,5 @@ __pycache__/
 .ipynb_checkpoints
 
 /.luarc.json
+
+**/*.quarto_ipynb

CITATION.cff

@@ -17,6 +17,8 @@ authors:
   family-names: "Barron"
 - given-names: "Zachary"
   family-names: "Deziel"
+- given-names: "Emmanuel"
+  family-names: "Mathot"
 - given-names: "Max"
   family-names: "Jones"
 - given-names: "Brianna"

_quarto.yml

@@ -31,6 +31,15 @@ website:
         text: Welcome
       - href: overview.qmd
         text: Overview Slides
+      # NOTE for @abarciauskas-bgse: proposed nav placement only.
+      # Placed before "Formats" so a reader can pick a format family before
+      # diving into a format page. Final position depends on the Track-3 site
+      # redesign - please confirm or move (and rename the path if needed).
+      - section: Choosing a Format
+        href: decision-framework/index.qmd
+        contents:
+          - decision-framework/index.qmd
+          - decision-framework/case-studies.qmd
       - section: Formats
         contents:
         - section: Cloud Optimized GeoTIFFs (COG)

_thankyous.qmd

@@ -10,4 +10,6 @@ This guide has been made possible through the support of:
 [![](./images/thankyous-devseed-logo.png){fig-alt="Development Seed Logo" fig-align="left" width=40%}](https://developmentseed.org)
 
 [![](./images/thankyous-cng-logo.png){fig-alt="Cloud-Native Geospatial Forum" fig-align="left" width=40%}](https://cloudnativegeo.org)
-:::
+:::
+
+The [Format Selection Decision Framework](./decision-framework/index.qmd) and its case studies were contributed from a study for the [Centre National d'Études Spatiales (CNES)](https://cnes.fr/), and use public [Copernicus](https://www.copernicus.eu/) and [NASA](https://www.earthdata.nasa.gov/) mission data.

decision-framework/case-studies.qmd

@@ -0,0 +1,147 @@
+---
+title: "Decision Framework Case Studies"
+subtitle: "Six products run through the framework, five of them backed by measurements"
+---
+
+Each case study below runs a real product through the [decision framework](index.qmd). It shows where the product lands and why. Five of the six are backed by **measured benchmark results**.
+
+Read each case in two parts:
+
+1. **The path through the spine**: the control fork, the topology gate, then intent, the object-size lever, and maturity.
+2. **What the measurement showed.**
+
+The examples use public Copernicus and NASA missions. The [CNES Datalake](https://cnes.fr/en/projects/centre-de-calcul) appears as one worked example of a tiered object store. It is not the only kind of backend, and its tier numbers are not a general rule.
+
+::: {.callout-important}
+## The recurring lesson
+
+**Going cloud-optimized is rarely smaller than the source you started from.**
+
+What it buys you is partial access, a spatial index, or a level of detail. Where two cloud-optimized candidates *do* differ in size, the cause is usually the **codec**, not the format.
+
+A second element explains much of the gap: **the extra bytes are the feature.** Overviews, levels of detail, and spatial indexes are structures the source file did not carry at all. They add volume, and they are exactly what buys fast display and cheap subsetting. The COG in case 1 is about 30 % larger than the delivered archive mostly because it carries overviews, and the row-group index in case 4 is the whole point of the conversion.
+:::
+
+::: {.callout-note}
+The numbers below are first measurements on a representative unit per mission, meaning one scene, one tile, or a few granules. They are illustrative, not a published benchmark. Two of the six cases have re-runs pending, and each one says so.
+
+They were produced with [`cng-formats-benchmark`](https://github.com/developmentseed/cng-formats-benchmark), an open-source harness that measures object-size distribution, write, read, and display per format. You can describe your own datasets in its config files and reproduce this shape of comparison on your own data. See [Optional: make it quantitative](index.qmd#optional-make-it-quantitative).
+:::
+
+## 1. Per-scene optical raster: COG vs GeoZarr for visualization and distribution
+
+**Product:** Sentinel-2 L2A. **Topology:** 2D raster per scene. **Source format:** geoTiff in a `.zip`. **Candidate set:** `{COG, GeoZarr}`.
+
+**[Intent](index.qmd#intent).** Distribution and visualization. Both are served per scene, which is why this case does not need a cube.
+
+**The path.** Existing data, and the producer operates the processing chain, so this is a free choice. The topology opens `{COG, GeoZarr}` and the [intent](index.qmd#intent) decides between them. Per-scene **distribution and visualization** favors COG. **Time-series analysis** favors a GeoZarr cube. The [grouping lever](index.qmd#the-object-size-and-grouping-lever) is COG internal tiling, or a scene-size Zarr shard.
+
+**Measured, one scene.** Both layouts cleared the warm tier. COG came to 2625.7 MiB in 28 objects. A scene-size GeoZarr shard came to 1975.6 MiB in 26 objects, about 25 % smaller. That gap is the **codec**, Deflate against ZSTD, not the format. Against the 2023.7 MiB delivered `.zip`, the GeoZarr store is about source size (−2 %) and the COG is larger (+30 %), because the COG carries overviews.
+
+**[Interpretation encoding](index.qmd#interpretation-encoding) is a real differentiator here.** The source reflectance is stored as scaled integers, and the nodata fill is left undeclared. COG puts the scale, the offset, and the nodata in side metadata that readers honor unevenly. Zarr v3 can carry all three as first-class array properties, using the `scale_offset` codec and a declared fill value, so every reader recovers the same physical value.
+
+**Where it lands.** COG per scene for distribution and visualization.
+
+For temporal analysis GeoZarr is the right target, but **the arm measured here does not yet deliver it.** What was written is a scene-size store: one scene, so there is no time dimension to exploit. It proves the container and the encoding, because it comes out at roughly source size and it carries the scale, offset, and nodata correctly. A time-series query against it still has to open one store per date, which is the same fan-out the COG layout has.
+
+The temporal benefit appears only once many scenes are concatenated along time into a single cube. That is the regime measured in [case 2](#case-time-cube), and it is a collection-level layout decision, not something a scene-size store can give you. Read the per-scene store as a building block and the cube as the product.
+
+The associated masking product, which is written as many small GeoTIFFs, is a clear anti-pattern to fix at the source.
+
+## 2. Raster time-cube: GeoZarr, with COG as the alternative {#case-time-cube}
+
+**Product:** Sentinel-1 RTC. **Topology:** 2D raster over time. **Source format:** geoTiff in a `.zip`. **Candidate set:** `{COG, GeoZarr}`.
+
+**[Intent](index.qmd#intent).** Advanced analysis and visualization lead, with distribution close behind but weighted a little lower. The analysis half is what separates this case from case 1: it is asked over time, not per scene, and that is what pulls the answer to a cube.
+
+**The path.** Existing data, reprocessable, so a free choice. A year of RTC over a region is naturally a `time × y × x × pol` cube. On a Zarr cube, a time-series slice or an area-of-interest subset is one range-addressable query. Across thousands of per-timestamp COGs, the same request becomes a large fan-out. Sharding reconciles fine-grained chunk access with the backend's preference for large objects.
+
+**Measured, one tile and confirmed at area scale.** COG came to 728 MiB per tile and a scene-size GeoZarr shard to 545 MiB. Both cleared the cold tier. GeoZarr is again about 25 % smaller, and again this is the codec: Deflate at 3.70x against ZSTD at 4.98x. The result held over a 16-tile block, at 16498 MiB for COG and 12344 MiB for GeoZarr. Against the 496 MiB delivered `.zip` both are larger, COG by 47 % and GeoZarr by 10 %. Cloud-optimized costs storage here, and buys the range access that the opaque zip cannot give at all.
+
+::: {.callout-warning}
+## A benchmark caveat, not a format verdict
+
+The GeoZarr store read and displayed much slower than the COG, but it was written **with no multiscale overview pyramid**. A store without multiscales is not a faithful GeoZarr, and the missing pyramid is the reason a tile server has to downsample full-resolution chunks on request. The slow read is also in large part a reader effect, from the library used rather than the format.
+
+Both issues are being fixed for the final campaign, so that display is compared on equal terms. Treat the timing numbers from this arm as provisional.
+:::
+
+## 3. Point cloud: netCDF to COPC
+
+**Product:** SWOT L2_HR Pixel Cloud. **Topology:** point cloud. **Source format:** netCDF. **Candidate set:** `{COPC, GeoParquet}`.
+
+**[Intent](index.qmd#intent).** Distribution, visualization, and data fusion. The fusion intent is the one that matters most here: these points get combined with other water products, so a reader has to recover the same physical value per point and subset the same area cheaply. That is what makes the missing spatial index in the source a real cost rather than an inconvenience.
+
+**The path.** Existing data, reprocessable, so a free choice. The point-cloud topology gives COPC, which is a clustered octree inside a LAZ file. The netCDF source has no spatial index and no level of detail, so reading a small geographic subset means scanning the whole granule.
+
+**Measured, content-complete, three granules.** A COPC carrying all per-point variables was about 16 % **smaller** than the source netCDF. This is the one clear size win among the cases, and it comes from the codec: LASzip at 3.72x against netCDF Deflate at 2.56x. Each granule cleared the tier on its own.
+
+The real gain is what the netCDF lacks:
+
+- A **spatial index**. A bounding-box query reads only the overlapping octree nodes.
+- A **level of detail**. Octree levels 0 to 11 are the point-cloud analogue of COG overviews.
+
+::: {.callout-caution}
+An earlier measurement on this product looked like a 97 % size saving. It had dropped most of the per-point variables. Size comparisons must be like for like on content.
+:::
+
+**Where it lands.** COPC. The cloud-optimized value is partial access plus level of detail, with a modest size win on top. GeoParquet is the alternative where columnar analytics dominate.
+
+## 4. Vector and tabular: shapefile to GeoParquet
+
+**Product:** SWOT L2_HR Lake Single Pass. **Topology:** vector and tabular. **Source format:** shapefile. **Candidate set:** `{GeoParquet, FlatGeobuf}`.
+
+**[Intent](index.qmd#intent).** Distribution, download, and data fusion. Note that visualization is *absent* from this mix, and download is present. That combination is why the answer is not automatically the columnar option: download rewards a single streamable file, which is FlatGeobuf's strength, while fusion rewards cheap joins on a fixed schema, which is GeoParquet's. The fusion weight is what settles it.
+
+**The path.** Existing data, reprocessable, so a free choice. This is a fixed-schema tabular product, with a lake id, a reach id, water-surface elevation, area, and more. That is the native fit for GeoParquet, with its columns, row groups, and spatial partitioning. A sparse Zarr store would add complexity with no gain.
+
+**Measured, two passes.** GeoParquet came to 17.8 MiB per pass, about 14 % **larger** than the 31.1 MiB zipped shapefile. This surprised reviewers, who expected a binary columnar format to be smaller. The likely cause is the default row-group codec, and it is under investigation.
+
+Each pass sits below the warm-tier floor. The lever is therefore **aggregation across passes**: concatenate first, then size the row groups. A one-for-one transcode of each pass would leave the object-size problem in place. The gain over the shapefile is the bounding-box row-group index, not the size.
+
+**Where it lands.** GeoParquet, with pass-level aggregation. FlatGeobuf is the single-file streaming alternative.
+
+## 5. Small N-D grid: the aggregation regime
+
+**Product:** SWOT L2_HR Raster. **Topology:** N-D grid. **Source format:** geoTiff. **Candidate set:** `{GeoZarr, COG}`.
+
+**[Intent](index.qmd#intent).** Distribution and download, with data fusion against the other SWOT products, and some visualization. This mix is provisional and still to be confirmed with the product owner.
+
+**The path.** Existing data, reprocessable, so a free choice. A 100 m raster granule covers a small tile, about 160 km across.
+
+**Measured, single variable, so not yet like for like.** On the primary water-surface-elevation variable, COG came to 4.3 MiB per granule and a GeoZarr shard to 0.8 MiB on average, because sharding fragments at this size. Both sit far below the warm-tier floor, so the lever is **aggregation**, and at this small size **COG is the cleaner carrier**: one tiled object that also holds overviews. An all-variable re-run is needed before quoting a size, for the same reason as case 3.
+
+The natural aggregation unit is a spatial mosaic, per grid zone and per orbit cycle. That turns a wall of tiny granules into one range-addressable object.
+
+**Where it lands.** COG for the small raster, plus collection-level aggregation. GeoZarr is the alternative to compare once the pyramid and all variables are in place.
+
+This case is the counterpart to cases 1 and 2. There, one large band already clears the tier, so you group *within* a product. Here the products are small, so you group *above* a single product.
+
+## 6. Existing archive to a virtual store
+
+**Topology:** any. **Candidate set:** a virtual Zarr store.
+
+This case has no benchmark. It illustrates the "existing data, cannot reprocess" branch of the control fork.
+
+An immutable archive, for example HDF5 or netCDF that you cannot reprocess, can be surfaced as a virtual Zarr store with [VirtualiZarr, Kerchunk, or Icechunk](../zarr/virtual-zarr.qmd). This gives range-based access without rewriting the data.
+
+The result is more limited than a native store. Chunk sizes are whatever the source chose, you cannot add overviews, and you inherit the source's codec. It is still much better than whole-file download, and it is often the only option available.
+
+This case is also where the ["unlock this"](index.qmd#dead-ends-are-unlock-this) framing matters. If the source cannot even be virtualized, name what is missing rather than declaring the case impossible.
+
+## Summary
+
+| Case | Topology | Lands on | Size against source | Main gain |
+| --- | --- | --- | --- | --- |
+| 1. Optical scene | 2D raster | COG per scene; a cube needs cross-scene aggregation | COG +30 %, GeoZarr −2 % | Partial access, encoding |
+| 2. SAR time-cube | Raster over time | GeoZarr v3 sharded | COG +47 %, GeoZarr +10 % | One query instead of a fan-out |
+| 3. Pixel cloud | Point cloud | COPC | −16 % | Spatial index, level of detail |
+| 4. Lake vector | Vector, tabular | GeoParquet, aggregated | +14 % | Row-group bbox index |
+| 5. Small raster grid | N-D grid | COG, plus aggregation | Re-run pending | Fewer, larger objects |
+| 6. Frozen archive | Any | Virtual Zarr | No rewrite | Range access with no reprocessing |
+
+Read the size column together with the recurring lesson at the top of this page. Only one case got smaller, and that case got smaller because of its codec.
+
+## Acknowledgements
+
+These case studies come from a study for the [Centre National d'Études Spatiales (CNES)](https://cnes.fr/), carried out by [Development Seed](https://developmentseed.org/). They use public Copernicus and NASA mission data. The measurements were produced with [`cng-formats-benchmark`](https://github.com/developmentseed/cng-formats-benchmark), which is maintained separately from this guide so that its methodological choices stay debatable in their own right. See [Our Supporters](../contributing.qmd#thank-you-to-our-supporters).