From 6b95927a9caf71fcba36fd7c0a05066349c0c639 Mon Sep 17 00:00:00 2001 From: Aimee Barciauskas Date: Tue, 8 Sep 2026 20:32:46 -0700 Subject: [PATCH 1/5] Minor wording changes --- decision-framework/case-studies.qmd | 14 ++++---- decision-framework/index.qmd | 56 ++++++++++++++--------------- 2 files changed, 35 insertions(+), 35 deletions(-) diff --git a/decision-framework/case-studies.qmd b/decision-framework/case-studies.qmd index ff31e4d..9664ba1 100644 --- a/decision-framework/case-studies.qmd +++ b/decision-framework/case-studies.qmd @@ -28,11 +28,11 @@ The examples use public Copernicus and NASA missions. The [CNES Datalake](https: Cloud-optimized formats provide partial access via byte-range indices into the data, accessible via addressable metadata. -**The extra bytes are the feature.** Overviews, levels of detail, and spatial indexes are structures the source file did not carry at all. They add volume, and they are exactly what buys fast display and cheap subsetting. The COG in case 1 is about 30% larger than the delivered archive mostly because it carries overviews, and the row-group index in case 4 is the whole point of the conversion. +**The extra bytes are the feature.** Overviews, levels of detail, and spatial indexes are structures the non-cloud-optimized versions do not carry at all (at least not by definition). They add volume, and they are exactly what buys fast display and cheap subsetting. The COG in case 1 is about 30% larger than the delivered archive mostly because it carries overviews, and the row-group index in case 4 is the whole point of the conversion. ::: ::: {.callout-note} -The numbers below are first measurements on a representative unit per mission, meaning one scene, one tile, or a few granules. They are illustrative, not a published benchmark. Two of the six cases have re-runs pending, and each one says so. +The numbers below are first measurements on a representative unit per mission, meaning one scene, one tile, or a few granules. They are illustrative, not a robust benchmark. Two of the six cases have re-runs pending. They were produced with [`cng-formats-benchmark`](https://github.com/developmentseed/cng-formats-benchmark), an open-source harness that measures object-size distribution, write, read, and display per format. You can describe your own datasets in its config files and reproduce this shape of comparison on your own data. See [Optional: make it quantitative](index.qmd#optional-make-it-quantitative). ::: @@ -49,9 +49,9 @@ They were produced with [`cng-formats-benchmark`](https://github.com/development Both intents are served per scene, which is why this case does not need a cube at all. -**Measured, one scene.** Both layouts cleared the [warm tier](../glossary.qmd#tiered-storage). COG came to 2625.7 MiB in 28 objects. A scene-size GeoZarr shard came to 1975.6 MiB in 26 objects, about 25 % smaller. That gap is the **codec**, Deflate against ZSTD, not the format. Against the 2023.7 MiB delivered `.zip`, the GeoZarr store is about source size (−2 %) and the COG is larger (+30 %), because the COG carries overviews. +**Measured, one scene.** Both layouts cleared the [warm tier](../glossary.qmd#tiered-storage). COG came to 2625.7 MiB in 28 objects. A scene-size GeoZarr shard came to 1975.6 MiB in 26 objects, about 25 % smaller. The difference comes from the compression algorithm, not the format. ZSTD was used for GeoZarr, and deflate was used for COG compression. Against the 2023.7 MiB delivered `.zip`, the GeoZarr store is about source size (−2 %) and the COG is larger (+30 %), because the COG carries overviews. -**[Interpretation encoding](index.qmd#interpretation-encoding) is a real differentiator here.** The source reflectance is stored as scaled integers, and the nodata fill is left undeclared. COG puts the scale, the offset, and the nodata in side metadata that readers honor unevenly. Zarr v3 can carry all three as first-class array properties, using the `scale_offset` codec and a declared fill value, so every reader recovers the same physical value. +**[Interpretation encoding](index.qmd#interpretation-encoding) is a real differentiator here.** The source reflectance is stored as scaled integers, and the nodata fill is left undeclared. COG puts the scale, the offset, and the nodata in a sidecar metadata file that readers honor unevenly. Zarr v3 can carry all three as first-class array properties, using the `scale_offset` codec and a declared fill value, so every reader recovers the same physical value. **Where it lands.** COG per scene for distribution and visualization. @@ -75,7 +75,7 @@ Sentinel-2 also has a second multiscale dimension worth naming: its bands are na The analysis half is what separates this case from case 1: it is asked over time, not per scene. A year of RTC over a region is naturally a `time × y × x × pol` cube, and on a Zarr cube a time-series slice or an area-of-interest subset is one range-addressable query. Across thousands of per-timestamp COGs the same request becomes a large fan-out. Sharding reconciles fine-grained chunk access with the object store's preference for large objects. -**Measured, one tile and confirmed at area scale.** COG came to 728 MiB per tile and a scene-size GeoZarr shard to 545 MiB. Both cleared the [cold tier](../glossary.qmd#tiered-storage). GeoZarr is again about 25 % smaller, and again this is the codec: Deflate at 3.70x against ZSTD at 4.98x. The result held over a 16-tile block, at 16498 MiB for COG and 12344 MiB for GeoZarr. Against the 496 MiB delivered `.zip` both are larger, COG by 47 % and GeoZarr by 10 %. Cloud-optimized costs storage here, and buys the range access that the opaque zip cannot give at all. +**Measured, one tile and confirmed at area scale.** COG came to 728 MiB per tile and a scene-size GeoZarr shard to 545 MiB. Both cleared the [cold tier](../glossary.qmd#tiered-storage). GeoZarr is again about 25 % smaller, and again this is the codec: Deflate at 3.70x against ZSTD at 4.98x. The result held over a 16-tile block, at 16498 MiB for COG and 12344 MiB for GeoZarr. Against the 496 MiB delivered `.zip` both are larger, COG by 47 % and GeoZarr by 10 %. Cloud-optimized costs storage here but buys the range access that the opaque zip cannot give at all. ## 3. Point cloud: netCDF to COPC @@ -118,7 +118,7 @@ The intent mix makes this case a genuine two-way contest, though not in the way - **Streaming is a visualization property, not a download one.** FlatGeobuf [streams](../flatgeobuf/intro.qmd#streaming-features-is-supported), so features render progressively as they arrive. Valuable, but visualization is absent from this mix. - **FlatGeobuf wins on whole-row spatial subsetting.** It is row-oriented with a Hilbert R-tree, so every attribute of the features in a bounding box is one contiguous read, where a [column-oriented](../geoparquet/index.qmd#column-oriented) store has to touch every column. A real access pattern, just not one this intent mix asks for. -At volume the scale question would matter too, because GeoParquet shards across [multiple files](../geoparquet/index.qmd#multi-file-support) while FlatGeobuf is a single file with a single index. These passes are tiny, so it does not bite here. +At volume the scale question would matter too, because GeoParquet shards across [multiple files](../geoparquet/index.qmd#multi-file-support) while FlatGeobuf is a single file with a single index. The SWOT L2_HR passes are tiny, so the scale question does not matter here. So GeoParquet wins on download *and* fusion. The product is fixed-schema tabular, with a lake id, a reach id, water-surface elevation, area and more, which is the native fit for its columns and row groups. A sparse Zarr store would add complexity with no gain. @@ -140,7 +140,7 @@ Each pass sits below the warm-tier floor. The lever is therefore **aggregation a - **[Intent](index.qmd#intent).** Distribution and download, with data fusion against the other SWOT products, and some visualization. This mix is provisional and still to be confirmed with the product owner. - **[Grouping lever](index.qmd#grouping-lever).** Aggregation *above* the product, not sharding within it. A 100 m granule covers a small tile, about 160 km across, so it is far below the warm-tier floor. -**Measured, single variable, so not yet like for like.** On the primary water-surface-elevation variable, COG came to 4.3 MiB per granule and a GeoZarr shard to 0.8 MiB on average, because sharding fragments at this size. Both sit far below the warm-tier floor, so the lever is **aggregation**, and at this small size **COG is the cleaner carrier**: one tiled object that also holds overviews. An all-variable re-run is needed before quoting a size, for the same reason as case 3. +**Measured, single variable, so not yet like for like.** On the primary water-surface-elevation variable, COG came to 4.3 MiB per file and a GeoZarr shard to 0.8 MiB on average, because sharding fragments at this size. Both sit far below the warm-tier floor, so the lever is **aggregation**, and at this small size **COG is the cleaner carrier**: one tiled object that also holds overviews. An all-variable re-run is needed before quoting a size, for the same reason as case 3. The natural aggregation unit is a spatial mosaic, per grid zone and per orbit cycle. That turns a wall of tiny granules into one range-addressable object. diff --git a/decision-framework/index.qmd b/decision-framework/index.qmd index e178cd6..008e963 100644 --- a/decision-framework/index.qmd +++ b/decision-framework/index.qmd @@ -10,11 +10,11 @@ This framework helps you go from *"I have geospatial data to publish or migrate" It is a guide, not an oracle: - **What you control** comes first: whether the data already exist, whether you may reprocess them, and whether a mandate fixes an archival format. This can settle the answer before format even comes up. -- The **shape of your data** narrows the field. +- The **shape of your data** narrows the set of candidate formats. - The **intended use** and a few practical levers rank the remaining options. - **You make the final call.** -The framework is provider-neutral. Where a concrete number helps, such as an object-size policy or a benchmark result, the number is given as a worked example and flagged as an example, not as a rule. +The framework is data- and cloud-provider-neutral. Within data and cloud-provider contexts, a rule may help make a decision. For example, an object-size policy or a benchmark result may influence the choice. These cases are given as worked examples, not rules. This is a first, deliberately simple version. It aligns on the questions to ask and the order to ask them in. Follow the links to a format page when you need more detail. @@ -36,10 +36,10 @@ The framework uses a few terms in a specific way. Each links to the section that | [**Archival-format mandate**](#control) | A requirement to maintain a specific archival format for preservation, such as HDF5, netCDF or GRIB, whatever cloud-optimized format you publish alongside it. | | [**Control**](#control) | Whether the choice is yours to make at all: do the data already exist, and may you reprocess them, technically and legally. This is Stage 1 of the framework, and it can end in "not yet possible". | | [**Ecosystem maturity**](#maturity) | How settled a format is, on an L1 to L5 scale from experimental to de-facto standard. It answers "safe default, or bet?". | -| [**Grouping lever**](#grouping-lever) | The knob each format gives you for grouping bytes into objects: COG internal tiling, Zarr v3 sharding, COPC octree nodes, GeoParquet row groups. | -| [**Intent**](#intent) | What the data are for: archive and download, processing, visualization, distribution, or data fusion. Most products serve several at once, in proportion. | +| [**Grouping lever**](#grouping-lever) | The knobs each format gives you for grouping bytes into data chunks and objects in cloud storage: COG internal tiling, Zarr v3 chunking and sharding, COPC octree nodes, GeoParquet row groups. | +| [**Intent**](#intent) | What the data are for: archive and download, processing, visualization, distribution, or data fusion. Most products serve several intentsat once. | | [**Interpretation encoding**](#interpretation-encoding) | Whether the format carries units, CRS, nodata and any scale and offset transform in-format, so every reader recovers the same physical value, or leans on a side catalog. | -| [**Object size**](#grouping-lever) | The size of the objects your layout actually writes to storage. Object stores reward a larger average object, and many apply a minimum size policy or sort objects into storage tiers. | +| [**Object size**](#grouping-lever) | The size of the objects your layout actually writes to storage. Object stores reward a larger average object and many apply a minimum size policy. | | [**Refinement**](#refinements) | A criterion applied *inside* a format family to rank its members. There are four: intent, the grouping lever, ecosystem maturity, and interpretation encoding. | | [**Topology**](#topology) | The shape of the data: 2D raster, N-D cubes, point cloud, or vector and tabular. Answering *what shape are the data?* selects a format family, not a format. | : {tbl-colwidths="[15,85]"} @@ -48,7 +48,7 @@ The framework uses a few terms in a specific way. Each links to the section that This guide already defines what makes a format cloud-optimized. The Welcome page states it as a four-part [cloud-optimized format pattern](../index.qmd#the-opportunity), and the glossary defines [cloud-optimized](../glossary.qmd#cloud-optimized) as the ability to read a meaningful *part* of a file without downloading *all* of it. -The framework does not restate that definition. It assumes it. Two consequences are worth naming here, because they decide whether a candidate belongs in the framework at all. They are entry conditions, not questions in the flowchart. +The framework starts from that definition Two consequences are worth naming here, because they decide whether a candidate belongs in the framework at all: **P1. The format must support partial and parallel reads over HTTP.** [Random access](../glossary.qmd#random-access) through [HTTP range requests](../glossary.qmd#http-range-request) is what makes a format usable from [object storage](../glossary.qmd#object-storage-cloud-storage). It depends on the rest of the pattern: [metadata](../glossary.qmd#metadata) in a consistent place that a reader can load in a single read, and then addressable [chunks](../glossary.qmd#chunk), internal tiling, or both. A reader that cannot locate the bytes it needs from the metadata alone cannot fetch only those bytes. @@ -66,8 +66,7 @@ is no format decision to make yet. **Stage 2 — topology and refinement: which format?** You reach this stage only when Stage 1 leaves you free to choose. Here the shape of the data selects a **format family**: the small group of cloud-optimized formats built for that -shape. A *family*, not a format. Two or three candidates normally come through, -and the refinements below rank them. +shape. The refinements below rank the options within a format family. The hand-off is worth stating outright, because the diagram can be read as if the shape of the data were the answer: @@ -142,7 +141,7 @@ flowchart TD #### How to tell which branch you are on -Whether you can reprocess or virtualize usually depends on the data and data project context. Answers to these questions are not judgement calls. +Whether you can reprocess or virtualize usually depends on the data and data project context. **You can reprocess** if you have the processor, the inputs, and the budget to re-run the production chain. Then format and layout are both free choices: go to @@ -184,7 +183,7 @@ The shape of the data selects a **format family**: the candidate set you refine | Data shape | Format family | Where each member wins | | --- | --- | --- | | 2D raster, per scene or per tile | [COG](../cloud-optimized-geotiffs/intro.qmd), [Zarr](../zarr/intro.qmd), [PMTiles](../pmtiles/intro.qmd) | COG for distribution and visualization; Zarr if the product is really a cube; PMTiles for pre-rendered tile delivery | -| N-D cubes, for example `time × y × x × band` | [GeoZarr](../zarr/intro.qmd), [COG](../cloud-optimized-geotiffs/intro.qmd), [Cloud-Optimized HDF/NetCDF](../cloud-optimized-netcdf4-hdf5/index.qmd) | GeoZarr for analysis across the cube; COG per time step if slices are read alone; cloud-optimized HDF/netCDF where an archival format is mandated or existing tooling must keep working | +| N-D cubes, for example `time × y × x × band` | [[Geo]Zarr](../zarr/intro.qmd), [COG](../cloud-optimized-geotiffs/intro.qmd), [Cloud-Optimized HDF/NetCDF](../cloud-optimized-netcdf4-hdf5/index.qmd) | GeoZarr for analysis across the cube; COG per time step if slices are read alone; cloud-optimized HDF/netCDF where an archival format is mandated or existing tooling must keep working | | Point cloud | [COPC](../copc/index.qmd), [GeoParquet](../geoparquet/index.qmd) | COPC for spatial subsetting and level of detail; GeoParquet if columnar analytics dominate | | Vector or tabular | [GeoParquet](../geoparquet/index.qmd), [FlatGeobuf](../flatgeobuf/intro.qmd), [PMTiles](../pmtiles/intro.qmd) | GeoParquet for joins and analytics; FlatGeobuf for streaming download; PMTiles for tiled web delivery | @@ -205,9 +204,9 @@ What are the data for? Archive and download, processing, visualization, distribu Most real datasets serve several intents at once. Weigh them in proportion rather than picking one. The [case studies](case-studies.qmd) state an intent mix for each product, which is what decides the ranking inside a candidate set. -### The object-size and grouping lever {#grouping-lever} +### The object size and grouping lever {#grouping-lever} -Every candidate format has a knob that sets how bytes are grouped into objects: +Every candidate format has a knob that sets how bytes are grouped into data chunks and chunks grouped into objects: | Format | Grouping lever | | --- | --- | @@ -218,12 +217,12 @@ Every candidate format has a knob that sets how bytes are grouped into objects: Tune the lever to your [object store's](../glossary.qmd#object-storage-cloud-storage) **object-size policy** (S3, Azure Blob Storage, Google Cloud Storage, or an on-premises equivalent). Object stores degrade when they must index very many tiny objects, so most reward a larger average object. -One worked example: the [CNES Datalake](https://cnes.fr/en/projects/centre-de-calcul) sorts data into [tiers](../glossary.qmd#tiered-storage) by mean object size, with about 32 MB or more for warm disk and about 100 MB or more for cold tape. This serves the general point: find out whether your storage has an object-size policy, and size the lever against it. Some providers publish a threshold, some only charge you as if your objects were larger, and some have no policy at all. +One worked example: the [CNES Datalake](https://cnes.fr/en/projects/centre-de-calcul) sorts data into [tiers](../glossary.qmd#tiered-storage) by mean object size, with about 32 MB or more for warm disk and about 100 MB or more for cold tape. This serves the general point: find out whether your storage has an object-size policy, and adjust your groupings accordingly. Some providers publish a threshold, some only charge you as if your objects were larger, and some have no policy at all. The lever works in both directions, and which direction you need depends on product size: - **Products that are already large.** Group *within* one product. Size the shard, the tile, or the row group so it clears the policy. -- **Products that are small.** Sharding within one granule only makes the fragmentation worse. Group *above* the product instead: mosaic or concatenate many granules into one range-addressable object. +- **Products that are small.** Sharding within one already small file only makes the fragmentation worse. Group *above* the product instead: mosaic or concatenate many small files into one range-addressable object. **Size and shape are two different questions.** How *big* the objects are is a question for your object store, answered above. How the bytes are arranged *inside* them @@ -244,7 +243,7 @@ at the [control](#control) question, because only reprocessing lets you choose i Ecosystem maturity may influence the final decision. You may need something production-grade that you can count on, or you may have the flexibility to be an early adopter. Maturity is what separates a safe default from a bet. -To keep that judgement explicit rather than impressionistic, place each candidate on a five-level scale: +To keep that judgement explicit rather than impressionistic, this framework assigns candidate formats on the following five-level scale: | Level | Meaning | | --- | --- | @@ -260,6 +259,7 @@ Where the candidates sit today: | --- | --- | --- | --- | | [COG](../cloud-optimized-geotiffs/intro.qmd) | De-facto standard, on a GeoTIFF base | GDAL, rio-tiler, TiTiler, every GIS | **L5** | | [GeoParquet](../geoparquet/index.qmd) | v1.x, stable, on an Apache Parquet base | GDAL, GeoPandas, DuckDB, Arrow | **L4** | +| [Zarr](../zarr/intro.qmd) | v2 and v3 stable | GDAL, zarr-python, zarrs | **L4** | | [COPC](../copc/index.qmd) | v1.0, stable, on a LAZ base | PDAL, untwine, QGIS, copc.js | **L4** | | [FlatGeobuf](../flatgeobuf/intro.qmd) | Stable, on a flatbuffers base | GDAL and OGR, QGIS, flatgeobuf.js | **L3** | | [GeoZarr](../zarr/intro.qmd) | Draft, in an OGC Standards Working Group | Emerging, mostly xarray-based stacks | **L2** | @@ -272,31 +272,31 @@ Where the candidates sit today: This distinction is easy to miss, and it decides how much risk you actually take on. -- **Zarr v3, the substrate, is dependable.** The core specification was finalized in 2024. [Sharding](#grouping-lever), the feature that matters most for object storage, is stable in `zarr-python` 3.x and readable by GDAL and xarray. Judged on its own, the Zarr v3 core behaves like an L4. +- **Zarr v3, the substrate of GeoZarr, is dependable.** The core specification was finalized in 2024. [Sharding](#grouping-lever), the feature that matters most for object storage, is stable in `zarr-python` 3.x and readable by GDAL and xarray. Judged on its own, the Zarr v3 core behaves like an L4. - **GeoZarr, the convention layer, is still being ratified.** It is what makes a generic [Zarr](../glossary.qmd#zarr) store *geospatial*: CRS, dimension semantics, and multiscale pyramids. It does for Zarr roughly what the GeoTIFF tags do for a plain TIFF. It is a draft in an OGC Standards Working Group, on the standards track but not yet an adopted OGC standard, and the published draft can lag behind active convention work. Check [geozarr.org](https://geozarr.org/) for the current status before you rely on a specific version. -So the L2 above belongs to the convention layer, not to Zarr. The leading reference implementations today, such as ESA's EOPF Sentinel output, are demonstrators rather than production guarantees. +So the L2 above belongs to the convention layer, not to Zarr. The leading reference implementations of GeoZarrtoday, such as ESA's EOPF Sentinel output, are demonstrators rather than production guarantees. #### Maturity is a warning, not a veto -An L2 candidate can still be the right answer. The N-D cubes branch is exactly that case: there is no mature alternative with the same topology, so a low level does not mean "choose something else". It means **plan for change**. Budget for convention churn, pin the versions you depend on, and expect the visualization story in particular to improve as multiscale tooling catches up. +An L2 candidate can still be the right answer. Multidimensional cubes have no mature candidate for the multidimensional topology. A low level does not mean "choose something else". It means **plan for change**. Budget for convention churn, pin the versions you depend on, and expect visualization tooling in particular to improve as tooling catches up. -Read maturity against your own tolerance. A one-off internal product and a mission archive with a twenty-year retention commitment can reasonably land on different levels from the same candidate set. +Many factors influence what maturity level you can tolerate. A one-off internal product and a mission archive with a twenty-year retention commitment can reasonably land on different levels from the same set of candidate formats. ### Interpretation encoding {#interpretation-encoding} -Does the format carry the information a reader needs to recover the same physical value, or does it lean on a side catalog? The information is units, the [CRS](../glossary.qmd#coordinate-reference-system-crs), the nodata fill, and any scale and offset transform. +Does the format carry the information a reader needs to recover the same physical value, or does it lean on a side catalog? Interpretation encoding includes units, the [CRS](../glossary.qmd#coordinate-reference-system-crs), the nodata fill value (see also [Virtualizarr's distinction of fill values](https://virtualizarr.readthedocs.io/en/latest/explanation/custom_parsers.html#fill-values)), and any scale and offset transform. -This axis separates the candidates more than people expect: +Interpretation encoding separates the candidates more than people expect: -- **Zarr v3 makes the information first-class.** With the `scale_offset` codec, a declared fill value, Zarr conventions for projection, CRS and multiscale, the transform is a first-class array property. Every reader recovers the same value. +- **Zarr v3 makes the information first-class.** With the `scale_offset` codec, a declared fill value, Zarr conventions for projection, CRS and multiscales, the transform is a first-class array property. Every reader recovers the same value. - **COG, GeoParquet, and COPC standardize the CRS** but leave the value transform to side metadata. Readers honor side metadata unevenly. -If your product stores scaled integers, or if an undeclared fill value would change a computed result, weigh this axis heavily. +If your product stores scaled integers, or if an undeclared fill value would change a computed result, weigh this aspect heavily. ## Cross-cutting considerations -These are not decision points. They apply whatever you choose. +These considerations do not impact what format you choose, but they do affect how you implement that format. ### Metadata and cataloging @@ -309,7 +309,7 @@ At a scale of tens of thousands to millions of files, collection-level discovery Some choices sit below the format and are foundational: - Base codecs, such as [ZSTD](../glossary.qmd#zstd) and [Deflate](../glossary.qmd#deflate) -- Data types and scale-offset encoding +- Data types, and scale-offset encoding - Sharding in Zarr v3 These are mature and in scope. The per-format pages cover them. More experimental optimizations, such as bit-rounding and advanced compression tricks, are out of scope here, or should be marked as immature when you use them. @@ -360,9 +360,9 @@ The idea is a **weighted mean of criteria**, each expressed as a number from 0 t | Measured, needs a benchmark run | **size**: output volume against the source | | | **performance**: measured read, write, and display, blended by the intent mix | -**Where the measured numbers come from.** The two measured criteria need a harness, and reproducibility is the whole point of quoting a number at all. [`cng-formats-benchmark`](https://github.com/developmentseed/cng-formats-benchmark) is the open-source (MIT) harness behind the numbers on the [Case Studies](case-studies.qmd) page. Per format and per dataset it measures the object-size distribution and tier fitness, write throughput, range-request-aware read latency, and tile-display latency. You describe your own datasets in config files, so no provider or dataset is baked into the code, and you run it on your own infrastructure against your own data. See the [documentation](https://developmentseed.github.io/cng-formats-benchmark/) and the [configuration guide](https://developmentseed.github.io/cng-formats-benchmark/configuration/). +**Where the measured numbers come from.** The two measured criteria need a benchmark, and reproducibility is the whole point of quoting a number at all. [`cng-formats-benchmark`](https://github.com/developmentseed/cng-formats-benchmark) is the open-source (MIT) harness behind the numbers on the [Case Studies](case-studies.qmd) page. Per format and per dataset it measures the object-size distribution and tier fitness, write throughput, range-request-aware read latency, and tile-display latency. You describe your own datasets in config files, so no provider or dataset is baked into the code, and you run it on your own infrastructure against your own data. See the [documentation](https://developmentseed.github.io/cng-formats-benchmark/) and the [configuration guide](https://developmentseed.github.io/cng-formats-benchmark/configuration/). -Rank the candidates and read the **margin**, which is the gap to the runner-up. A small margin is an honest signal that a benchmark, or a review of your weights, is what actually decides it. There are **no verdicts and no probability readout**. You get a score, a ranking, and a margin. +You can use this framework to rank the candidates and evaluate the **margin** between the winner and the runner-up. A small margin is an honest signal that a benchmark, or a review of your weights, is what actually decides it. There are **no verdicts and no probability readout**. You get a score, a ranking, and a margin. Two points are easy to get wrong: @@ -381,4 +381,4 @@ One lesson recurs often enough to state up front: **going cloud-optimized is rar This framework was developed as part of a study for the [Centre National d'Études Spatiales (CNES)](https://cnes.fr/), carried out by [Development Seed](https://developmentseed.org/). The case studies use public Copernicus and NASA missions, and the measurements come from that study's benchmark work. -The framework builds on earlier work in this guide supported by [NASA IMPACT](https://impact.earthdata.nasa.gov/). See [Our Supporters](../contributing.qmd#thank-you-to-our-supporters). +The framework builds on earlier work in this guide supported by [NASA IMPACT](https://impact.earthdata.nasa.gov/) (now [NASA ODSI](https://www.nasa.gov/marshall/marshall-space-flight-missions/office-of-data-science-and-informatics-odsi/)). See [Our Supporters](../contributing.qmd#thank-you-to-our-supporters). From 22dec0b2418bf5e6502fdc50b7bc79baa7431d28 Mon Sep 17 00:00:00 2001 From: Aimee Barciauskas Date: Wed, 9 Sep 2026 07:58:23 -0700 Subject: [PATCH 2/5] More changes --- decision-framework/index.qmd | 30 +++++++++++------------------- 1 file changed, 11 insertions(+), 19 deletions(-) diff --git a/decision-framework/index.qmd b/decision-framework/index.qmd index 008e963..3b1ac25 100644 --- a/decision-framework/index.qmd +++ b/decision-framework/index.qmd @@ -7,7 +7,7 @@ subtitle: "How to go from data you must publish to a short list of cloud-optimiz This framework helps you go from *"I have geospatial data to publish or migrate"* to a **short list of cloud-optimized format candidates**, and to the **layout choices** that make each candidate work. -It is a guide, not an oracle: +It is a guide to help make informed decisions, and does not prescribe a single correct format. - **What you control** comes first: whether the data already exist, whether you may reprocess them, and whether a mandate fixes an archival format. This can settle the answer before format even comes up. - The **shape of your data** narrows the set of candidate formats. @@ -48,7 +48,7 @@ The framework uses a few terms in a specific way. Each links to the section that This guide already defines what makes a format cloud-optimized. The Welcome page states it as a four-part [cloud-optimized format pattern](../index.qmd#the-opportunity), and the glossary defines [cloud-optimized](../glossary.qmd#cloud-optimized) as the ability to read a meaningful *part* of a file without downloading *all* of it. -The framework starts from that definition Two consequences are worth naming here, because they decide whether a candidate belongs in the framework at all: +The framework starts from the definition on the Welcome page. Two requirements are worth explaining in more detail, because they decide whether a candidate belongs in the framework at all: **P1. The format must support partial and parallel reads over HTTP.** [Random access](../glossary.qmd#random-access) through [HTTP range requests](../glossary.qmd#http-range-request) is what makes a format usable from [object storage](../glossary.qmd#object-storage-cloud-storage). It depends on the rest of the pattern: [metadata](../glossary.qmd#metadata) in a consistent place that a reader can load in a single read, and then addressable [chunks](../glossary.qmd#chunk), internal tiling, or both. A reader that cannot locate the bytes it needs from the metadata alone cannot fetch only those bytes. @@ -56,7 +56,7 @@ The framework starts from that definition Two consequences are worth naming here ## The two stages -The framework runs in **two stages**, and the topology question marks the hand-off between them. +The framework has **two stages**: **Stage 1 — control: do you have a choice?** Do the data already exist? If they exist, are you able (technically, legally) to reprocess them? This stage can end @@ -68,15 +68,6 @@ when Stage 1 leaves you free to choose. Here the shape of the data selects a **format family**: the small group of cloud-optimized formats built for that shape. The refinements below rank the options within a format family. -The hand-off is worth stating outright, because the diagram can be read as if -the shape of the data were the answer: - -| Before the topology question | After it | -| --- | --- | -| Do I have a cloud-optimized target at all? | Which of these targets suits me best? | -| The outcome can be "not yet possible" | Every remaining candidate works; you are ranking | -| Decided by ownership and reprocessing rights | Decided by intent, layout, maturity, encoding | - Four refinements then rank the candidates: **intent**, the **object-size and grouping lever**, **ecosystem maturity**, and **interpretation encoding**. The last one deserves a close look, because the formats differ on it more than @@ -129,7 +120,9 @@ flowchart TD **No existing data.** You are designing a new product, so you can choose freely. Go to the [topology question](#topology). -**If an archival-format mandate applies,** it adds a second job, and it does not remove your freedom to choose how readers get access. Applying cloud-optimized practice to the archive itself is not optional: whatever the mandated format, make it as range-request-friendly as it can be. What is genuinely optional is what you do beyond that. The stronger, recommended option is to also publish a full cloud-optimized copy for readers, since a native copy gives real range access rather than the archive's own chunking. A lighter option is to skip the second copy and instead wrap the optimized archive in a [virtual Zarr](../zarr/virtual-zarr.qmd) layer: cheaper to maintain, but you inherit the archive's chunk shape, the same limit as [existing data you cannot reprocess](#control) below. Where the mandated format is HDF5 or netCDF, this guide has [specific guidance](../cloud-optimized-netcdf4-hdf5/index.qmd) on consolidated metadata and chunk sizing. For other archival formats the principle holds even where the guide has no page: optimize what you must keep, then choose deliberately how far you go for readers. +**If an archival-format mandate applies,** it adds a second job, and it does not remove your freedom to choose how readers get access. Whatever the mandated format, make it as range-request-friendly as it can be. Where the mandated format is HDF5 or netCDF, this guide has [specific guidance](../cloud-optimized-netcdf4-hdf5/index.qmd) on consolidated metadata and chunk sizing. What is genuinely optional is what you do beyond the archival format. + +You can wrap the optimized archive in a [virtual Zarr](../zarr/virtual-zarr.qmd) layer: cheaper to maintain, but you inherit the archive's chunk shape, the same limit as [existing data you cannot reprocess](#control) below. The stronger, recommended option is to also publish a cloud-optimized copy for readers, since a cloud-optimized copy is an opportunity to provide access beyond the archive's native chunking. **Existing data you can reprocess.** You control the production chain, so you can also choose freely. Go to the [topology question](#topology). @@ -176,7 +169,7 @@ technically and legally free to change format and still be held by what its user tools expect. See [Existing users and their tools](#existing-users). -### Topology: what shape are the data? {#topology} +## Topology: what shape are the data? {#topology} The shape of the data selects a **format family**: the candidate set you refine within. It does not select a format. [Intent](#intent) is what decides between the members. @@ -192,7 +185,7 @@ The question is about topology, not about the format the data arrives in. A netC ## Refinements {#refinements} -Apply these within the candidate set your topology opened. +These refinements help rank candidate formats within a family. ### Intent {#intent} @@ -217,12 +210,11 @@ Every candidate format has a knob that sets how bytes are grouped into data chun Tune the lever to your [object store's](../glossary.qmd#object-storage-cloud-storage) **object-size policy** (S3, Azure Blob Storage, Google Cloud Storage, or an on-premises equivalent). Object stores degrade when they must index very many tiny objects, so most reward a larger average object. -One worked example: the [CNES Datalake](https://cnes.fr/en/projects/centre-de-calcul) sorts data into [tiers](../glossary.qmd#tiered-storage) by mean object size, with about 32 MB or more for warm disk and about 100 MB or more for cold tape. This serves the general point: find out whether your storage has an object-size policy, and adjust your groupings accordingly. Some providers publish a threshold, some only charge you as if your objects were larger, and some have no policy at all. +One worked example: the [CNES Datalake](https://cnes.fr/en/projects/centre-de-calcul) sorts data into [tiers](../glossary.qmd#tiered-storage) by mean object size, with about 32 MB or more for warm disk and about 100 MB or more for cold tape. This serves the general point: find out whether your storage has an object-size policy, and adjust your groupings accordingly. -The lever works in both directions, and which direction you need depends on product size: +Some providers publish a threshold, some only charge you as if your objects were larger, and some have no policy at all. -- **Products that are already large.** Group *within* one product. Size the shard, the tile, or the row group so it clears the policy. -- **Products that are small.** Sharding within one already small file only makes the fragmentation worse. Group *above* the product instead: mosaic or concatenate many small files into one range-addressable object. +Objects can be too large (readers have to download and decompress more data than they need) or too small (readers make many small requests and pay the network latency for each). **Size and shape are two different questions.** How *big* the objects are is a question for your object store, answered above. How the bytes are arranged *inside* them From a757b2fbc4492a32fcd999809c825a27164674ab Mon Sep 17 00:00:00 2001 From: Aimee Barciauskas Date: Wed, 9 Sep 2026 13:34:06 -0700 Subject: [PATCH 3/5] Add more wording changes --- decision-framework/index.qmd | 65 +++++++++++++----------------------- glossary.qmd | 4 +++ 2 files changed, 28 insertions(+), 41 deletions(-) diff --git a/decision-framework/index.qmd b/decision-framework/index.qmd index 3b1ac25..f70feab 100644 --- a/decision-framework/index.qmd +++ b/decision-framework/index.qmd @@ -35,7 +35,7 @@ The framework uses a few terms in a specific way. Each links to the section that | [**Access pattern**](../glossary.qmd#access-pattern) | What subsets readers will request: a time series at one point, a spatial window at one date, a full scan. Intent implies an access pattern, and your layout either serves it or fights it. Defined in the guide's glossary; the layout consequences are in [the grouping lever](#grouping-lever). | | [**Archival-format mandate**](#control) | A requirement to maintain a specific archival format for preservation, such as HDF5, netCDF or GRIB, whatever cloud-optimized format you publish alongside it. | | [**Control**](#control) | Whether the choice is yours to make at all: do the data already exist, and may you reprocess them, technically and legally. This is Stage 1 of the framework, and it can end in "not yet possible". | -| [**Ecosystem maturity**](#maturity) | How settled a format is, on an L1 to L5 scale from experimental to de-facto standard. It answers "safe default, or bet?". | +| [**Ecosystem maturity**](#maturity) | How settled a format is, on an L1 to L5 scale from experimental to de-facto standard (see below). It answers "safe default, or bet?". | | [**Grouping lever**](#grouping-lever) | The knobs each format gives you for grouping bytes into data chunks and objects in cloud storage: COG internal tiling, Zarr v3 chunking and sharding, COPC octree nodes, GeoParquet row groups. | | [**Intent**](#intent) | What the data are for: archive and download, processing, visualization, distribution, or data fusion. Most products serve several intentsat once. | | [**Interpretation encoding**](#interpretation-encoding) | Whether the format carries units, CRS, nodata and any scale and offset transform in-format, so every reader recovers the same physical value, or leans on a side catalog. | @@ -116,7 +116,7 @@ flowchart TD ``` -### The control fork {#control} +## The control fork {#control} **No existing data.** You are designing a new product, so you can choose freely. Go to the [topology question](#topology). @@ -132,7 +132,7 @@ You can wrap the optimized archive in a [virtual Zarr](../zarr/virtual-zarr.qmd) **Neither.** Some sources cannot be repackaged or virtualized today. Examples are HDF4, encrypted files, and a single large blob with no internal structure, whether or not it's compressed. See [Not yet possible](#not-yet-possible). -#### How to tell which branch you are on +### How to tell which branch you are on Whether you can reprocess or virtualize usually depends on the data and data project context. @@ -171,7 +171,7 @@ tools expect. See [Existing users and their tools](#existing-users). ## Topology: what shape are the data? {#topology} -The shape of the data selects a **format family**: the candidate set you refine within. It does not select a format. [Intent](#intent) is what decides between the members. +The shape of the data selects a **format family**: the set of format candidates the refinement process narrows down. [Intent](#intent) is what decides between the members. | Data shape | Format family | Where each member wins | | --- | --- | --- | @@ -195,7 +195,7 @@ What are the data for? Archive and download, processing, visualization, distribu - **On-the-fly and batched processing pull the layout in opposite directions.** On-the-fly access rewards small chunks, fine overviews, and cache locality: a reader fetches a little and wants it back fast. Batched processing rewards large shards or row groups, so that a bulk scan becomes a few large contiguous reads that many workers can run in parallel, instead of thousands of small scattered ones. - **Data fusion raises the stakes on encoding.** Combining products from different missions or processors means every input must be read back as the same physical value, and subsets from each must line up. Fusion is a form of processing, but it is worth naming separately: it weights [interpretation encoding](#interpretation-encoding) and consistent grids far higher than single-product analysis does, and it usually rewards a tabular or columnar target where joins are cheap. -Most real datasets serve several intents at once. Weigh them in proportion rather than picking one. The [case studies](case-studies.qmd) state an intent mix for each product, which is what decides the ranking inside a candidate set. +Most real datasets serve several intents at once. However, the intents often carry different weight in the decision process. The [case studies](case-studies.qmd) state an intent mix for each product, which is what decides the ranking inside a candidate set. ### The object size and grouping lever {#grouping-lever} @@ -204,28 +204,28 @@ Every candidate format has a knob that sets how bytes are grouped into data chun | Format | Grouping lever | | --- | --- | | COG | internal tiling and overviews | -| Zarr v3 | chunks within shards ([sharding](../glossary.qmd#chunk-size)) | +| Zarr v3 | chunks and shards ([chunk](../glossary.qmd#chunk) and [shard](../glossary.qmd#shard)) | | COPC | octree nodes | | GeoParquet | row groups, plus partitioning | -Tune the lever to your [object store's](../glossary.qmd#object-storage-cloud-storage) **object-size policy** (S3, Azure Blob Storage, Google Cloud Storage, or an on-premises equivalent). Object stores degrade when they must index very many tiny objects, so most reward a larger average object. +Tune the lever to your [object store's](../glossary.qmd#object-storage-cloud-storage) **object-size policy** (S3, Azure Blob Storage, Google Cloud Storage, or an on-premises equivalent). Object stores degrade when they must index very many tiny objects, so most reward a larger average object. -One worked example: the [CNES Datalake](https://cnes.fr/en/projects/centre-de-calcul) sorts data into [tiers](../glossary.qmd#tiered-storage) by mean object size, with about 32 MB or more for warm disk and about 100 MB or more for cold tape. This serves the general point: find out whether your storage has an object-size policy, and adjust your groupings accordingly. +In general, when it comes to reading from object storage, objects can be too large (readers have to download and decompress more data than they need) or too small (readers make many small requests and pay the network latency for each). See also: [the datacube guid on tiny chunks](https://developmentseed.org/datacube-guide/latest/worst-practices/tiny-chunks.html) and [massive chunks](https://developmentseed.org/datacube-guide/latest/worst-practices/massive-chunks.html). -Some providers publish a threshold, some only charge you as if your objects were larger, and some have no policy at all. +One example: the [CNES Datalake](https://cnes.fr/en/projects/centre-de-calcul) sorts data into [tiers](../glossary.qmd#tiered-storage) by mean object size, with about 32 MB or more for warm disk and about 100 MB or more for cold tape. This serves the general point: find out whether your storage has an object-size policy, and adjust your groupings accordingly. -Objects can be too large (readers have to download and decompress more data than they need) or too small (readers make many small requests and pay the network latency for each). +Some providers publish a threshold, some only charge you as if your objects were larger, and some have no policy at all. **Size and shape are two different questions.** How *big* the objects are is a question for your object store, answered above. How the bytes are arranged *inside* them (chunk shape, and the order of the dimensions) is an **access-pattern** question, -and the object store has no opinion on it. A `time × y × x` cube chunked long in time +and an object store provider has no opinion about the shape. A `time × y × x` cube chunked long in time serves a time series at one point cheaply and a full-scene map expensively; chunked wide in `y` and `x` it does the opposite. This guide works it through, with a diagram, in [chunk shape vs chunk size](../cloud-optimized-netcdf4-hdf5/index.qmd#chunk-shape-vs-chunk-size). Shape rarely decides *which* format you pick, because every candidate in a family lets -you set it: a COG chooses its tile dimensions, Zarr its chunk shape and dimension +you set the shape: a COG chooses its tile dimensions, Zarr its chunk shape and dimension order, GeoParquet its row-group size and row ordering. What shape decides is how well the format you picked performs, not which one you pick. And it is mostly settled back at the [control](#control) question, because only reprocessing lets you choose it freely. @@ -239,7 +239,7 @@ To keep that judgement explicit rather than impressionistic, this framework assi | Level | Meaning | | --- | --- | -| **L5** | De-facto standard, universal tooling, no specification risk | +| **L5** | De-facto standard: universal tooling, no specification risk | | **L4** | Established: released spec, broad independent tooling, in production use | | **L3** | Stable: released spec, multiple implementations, growing tooling | | **L2** | Emerging: spec stabilizing, reference implementations, early adopters | @@ -267,11 +267,11 @@ This distinction is easy to miss, and it decides how much risk you actually take - **Zarr v3, the substrate of GeoZarr, is dependable.** The core specification was finalized in 2024. [Sharding](#grouping-lever), the feature that matters most for object storage, is stable in `zarr-python` 3.x and readable by GDAL and xarray. Judged on its own, the Zarr v3 core behaves like an L4. - **GeoZarr, the convention layer, is still being ratified.** It is what makes a generic [Zarr](../glossary.qmd#zarr) store *geospatial*: CRS, dimension semantics, and multiscale pyramids. It does for Zarr roughly what the GeoTIFF tags do for a plain TIFF. It is a draft in an OGC Standards Working Group, on the standards track but not yet an adopted OGC standard, and the published draft can lag behind active convention work. Check [geozarr.org](https://geozarr.org/) for the current status before you rely on a specific version. -So the L2 above belongs to the convention layer, not to Zarr. The leading reference implementations of GeoZarrtoday, such as ESA's EOPF Sentinel output, are demonstrators rather than production guarantees. +So the L2 above belongs to the convention layer, not to Zarr. The leading reference implementations of GeoZarr today, such as ESA's EOPF Sentinel output, are demonstrators rather than production guarantees. #### Maturity is a warning, not a veto -An L2 candidate can still be the right answer. Multidimensional cubes have no mature candidate for the multidimensional topology. A low level does not mean "choose something else". It means **plan for change**. Budget for convention churn, pin the versions you depend on, and expect visualization tooling in particular to improve as tooling catches up. +An L2 candidate can still be the right answer. Multidimensional cubes have no mature format candidate. A low level does not mean "choose something else". It means **plan for change**. Budget for convention churn, pin the versions you depend on, and expect visualization tooling in particular to improve as tooling catches up. Many factors influence what maturity level you can tolerate. A one-off internal product and a mission archive with a twenty-year retention commitment can reasonably land on different levels from the same set of candidate formats. @@ -304,15 +304,17 @@ Some choices sit below the format and are foundational: - Data types, and scale-offset encoding - Sharding in Zarr v3 -These are mature and in scope. The per-format pages cover them. More experimental optimizations, such as bit-rounding and advanced compression tricks, are out of scope here, or should be marked as immature when you use them. +These are mature and in scope. The per-format pages cover them. More experimental optimizations, such as bit-rounding and advanced compression tricks, are out of scope at this time. Formats also differ in which data types they support, and in whether types can be mixed within one file, column, or schema. Those specifics live on each format's own page and are worth checking before committing to a candidate; they can rule one out as surely as topology or intent. ### Multiscale and native mixed resolution -A multiscale pyramid does not have to come from one source resolution downsampled by a fixed factor. The common case is **overviews**, also called [pyramids](../glossary.qmd#overviews): downsampled copies of the same band, generated at a uniform decimation so a reader can zoom out without reading and resampling full-resolution chunks. COG overviews, COPC's octree levels of detail, and most GeoZarr pyramids work this way, and it is already covered above under [ecosystem maturity](#maturity) and the [grouping lever](#grouping-lever). +Providing downsampled levels of original data in the source resolution is a common pattern for providing good performance, usually for visualization. A common example is COG overviews, which provide tiles at standard resolutions for mapping clients. A multiscale pyramid is a more general concept: a set of levels of detail, each with its own resolution, that can be used for visualization or analysis. + +Often these multiscale pyramids are generated at a uniform decimation so a reader can zoom out without reading and resampling full-resolution chunks. COG overviews, COPC's octree levels of detail, and most GeoZarr pyramids work this way, and it is already covered above under [ecosystem maturity](#maturity) and the [grouping lever](#grouping-lever). -The same structure can also host bands that were never derived from each other and were simply captured at different native resolutions, as long as the steps between levels are chosen to land on each native resolution rather than a fixed factor. Sentinel-2 is the reference case, worked through in [case 1](case-studies.qmd) of the case studies: the 10 m, 20 m, and 60 m bands from the same sensor fold into one pyramid by aligning the decimation to each step (10 m to 20 m halves the resolution, 20 m to 60 m divides it by three), and the pyramid can continue past 60 m into ordinary overviews using the same mechanism. The [Zarr multiscales convention](https://github.com/zarr-conventions/multiscales) is built for exactly this: scale factors between levels are not required to be uniform, and its own reference examples use this Sentinel-2 layout. +How these multiscale pyramids are generated can vary: Sometimes, data is provided at varying resolution where lower resolution data is not derived from the higher resolution data. Sentinel-2 is an example, worked through in [case 1](case-studies.qmd) of the case studies: Sentinel-2 collects data in 10m, 20m, and 60m bands from the same sensor. These bands can become a pyramid by aligning the decimation to each step (10m to 20m halves the resolution, 20m to 60m divides it by three), and the pyramid can continue past 60 m into ordinary overviews using the same mechanism. [Zarr multiscales convention](https://github.com/zarr-conventions/multiscales) allows for non-uniform scale factors. This still interacts with the [grouping lever](#grouping-lever), since each native resolution needs its own tiling or chunk shape, and with [interpretation encoding](#interpretation-encoding), since each band needs its own geotransform. What changes from the plain-overview case is that these become refinements within one multiscale structure. @@ -334,34 +336,15 @@ In the short term the answer is usually the same: publish the cloud-optimized fo When a source cannot be repackaged or virtualized today, the honest answer is *"not yet implemented"*, not *"impossible"*. The blocker is usually a missing parser, an unsupported codec, or source files with no internal chunking at all. Every one of those can change. -The precedent is HDF5 itself: once the standard example of a format that could not be read efficiently from object storage, until a change in write strategy and reader tooling made it viable, see [Cloud-Optimized HDF/NetCDF](../cloud-optimized-netcdf4-hdf5/index.qmd).The reader half is still moving: switching xarray's default cache for the h5netcdf engine improves dataset opening ([xarray #11216](https://github.com/pydata/xarray/pull/11216), measured by its authors on NISAR data). Formats move, and so do the libraries. +The precedent is HDF5 itself: once the standard example of a format that could not be read efficiently from object storage, until a change in write strategy and reader tooling made it viable, see [Cloud-Optimized HDF/NetCDF](../cloud-optimized-netcdf4-hdf5/index.qmd). The reader half is still moving: switching xarray's default cache for the h5netcdf engine improves dataset opening ([xarray #11216](https://github.com/pydata/xarray/pull/11216), measured by its authors on NISAR data). Formats move, and so do the libraries. -Frame the branch that way, and route the reader back to the options: reprocess if you can get the chain changed, virtualize if a reader exists, or convert on ingest. Say what is missing, so somebody can unlock it. +Not yet possible does not mean never possible. Re-evaluate the options to pinpoint what needs to change to make cloud-optimized access possible. This could mean revaluating reprocessing, conversion and virtualization. ## Optional: make it quantitative -The two stages above get you to a short list. When two or three candidates remain and the choice is close, a transparent score can make the trade-off explicit rather than a matter of taste. This section is **optional**. The framework stands on its own without it. - -The idea is a **weighted mean of criteria**, each expressed as a number from 0 to 1. It is framework-first and metrics-optional: qualitative terms alone already give a ranking, and a benchmark run sharpens the same ranking. - -| Group | Criterion | -| --- | --- | -| Qualitative, no benchmark needed | **use-case fit**: your intent mix, applied to a per-format suitability matrix | -| | **maturity**: the L1 to L5 level above, mapped to a utility (`level / 5`) | -| | **interpretation encoding**: how much the format carries in-format, mapped to a utility | -| Measured, needs a benchmark run | **size**: output volume against the source | -| | **performance**: measured read, write, and display, blended by the intent mix | - -**Where the measured numbers come from.** The two measured criteria need a benchmark, and reproducibility is the whole point of quoting a number at all. [`cng-formats-benchmark`](https://github.com/developmentseed/cng-formats-benchmark) is the open-source (MIT) harness behind the numbers on the [Case Studies](case-studies.qmd) page. Per format and per dataset it measures the object-size distribution and tier fitness, write throughput, range-request-aware read latency, and tile-display latency. You describe your own datasets in config files, so no provider or dataset is baked into the code, and you run it on your own infrastructure against your own data. See the [documentation](https://developmentseed.github.io/cng-formats-benchmark/) and the [configuration guide](https://developmentseed.github.io/cng-formats-benchmark/configuration/). - -You can use this framework to rank the candidates and evaluate the **margin** between the winner and the runner-up. A small margin is an honest signal that a benchmark, or a review of your weights, is what actually decides it. There are **no verdicts and no probability readout**. You get a score, a ranking, and a margin. - -Two points are easy to get wrong: - -- **Object-size and tier fit is reported information, not a scored penalty.** Report whether a layout clears a given tier, the mean and median object size, and a small-file warning **beside** the score, never inside it. An object store's policy is a property of that store, not of the format, so you must weigh it against your own infrastructure. -- **Intent weights are your input.** The suitability matrix is a shared expert reference on each format's strengths, and it is useful on its own. The intent mix and the criterion weights belong to your context. +The two stages above get you to a short list. When two or three candidates remain and the choice is close, a transparent score can make the trade-off explicit and quantitative, reducing bias. -The convincing artifact is never a single score. It is a sensitivity sweep that shows the ranking holds across a range of weights. That is also the honest reason this section stays short and links out: a score you cannot reproduce on your own data is not worth much, so the harness matters more than any number this page could print. +[`cng-formats-benchmark`](https://github.com/developmentseed/cng-formats-benchmark) is an open-source (MIT) harness behind the numbers on the [Case Studies](case-studies.qmd) page. Per format and per dataset it measures the object-size distribution and tier fitness, write throughput, range-request-aware read latency, and tile-display latency. You describe your own datasets in config files, so no provider or dataset is baked into the code, and you run it on your own infrastructure against your own data. See the [documentation](https://developmentseed.github.io/cng-formats-benchmark/) and the [configuration guide](https://developmentseed.github.io/cng-formats-benchmark/configuration/). ## Worked examples diff --git a/glossary.qmd b/glossary.qmd index 89aeafe..0d7b9d3 100644 --- a/glossary.qmd +++ b/glossary.qmd @@ -318,6 +318,10 @@ An R library for using and managing [vector data](#vector-data), organized aroun A [vector data](#vector-data) file format. There are [many reasons](http://switchfromshapefile.org/) to no longer use Shapefile. +#### Shard + +The definition of shard here is specific to [Zarr](#zarr). A shard is a single object in object storage that contains multiple [chunks](#chunk) of data. See also: [Zarr sharding](https://zarr.readthedocs.io/en/latest/user-guide/performance/#sharding). + #### Space-filling curve An algorithm that translates two- or n- dimensional data into one-dimensional data. In practice, this is used as part of [spatial indexes](#spatial-index) to group [vector geometries](#vector-data) nearby within a file according to their two-dimensional location. From e287e2c3d8b4c5ffc52d768c2e5a7f603f5a8da6 Mon Sep 17 00:00:00 2001 From: "Alex I. Mandel" Date: Fri, 25 Sep 2026 09:40:39 -0700 Subject: [PATCH 4/5] typo Co-authored-by: Emmanuel Mathot --- decision-framework/index.qmd | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/decision-framework/index.qmd b/decision-framework/index.qmd index c00da60..ae77ad2 100644 --- a/decision-framework/index.qmd +++ b/decision-framework/index.qmd @@ -37,7 +37,7 @@ The framework uses a few terms in a specific way. Each links to the section that | [**Control**](#control) | Whether the choice is yours to make at all: do the data already exist, and may you reprocess them, technically and legally. This is Stage 1 of the framework, and it can end in "not yet possible". | | [**Ecosystem maturity**](#maturity) | How settled a format is, on an L1 to L5 scale from experimental to de-facto standard (see below). It answers "safe default, or bet?". | | [**Grouping lever**](#grouping-lever) | The knobs each format gives you for grouping bytes into data chunks and objects in cloud storage: COG internal tiling, Zarr v3 chunking and sharding, COPC octree nodes, GeoParquet row groups. | -| [**Intent**](#intent) | What the data are for: archive and download, processing, visualization, distribution, or data fusion. Most products serve several intentsat once. | +| [**Intent**](#intent) | What the data are for: archive and download, processing, visualization, distribution, or data fusion. Most products serve several intents at once. | | [**Interpretation encoding**](#interpretation-encoding) | Whether the format carries units, CRS, nodata and any scale and offset transform in-format, so every reader recovers the same physical value, or leans on a side catalog. | | [**Object size**](#grouping-lever) | The size of the objects your layout actually writes to storage. Object stores reward a larger average object and many apply a minimum size policy. | | [**Refinement**](#refinements) | A criterion applied *inside* a format family to rank its members. There are four: intent, the grouping lever, ecosystem maturity, and interpretation encoding. | From 5b208537c4863bf6cd59a121fbdb957588afd338 Mon Sep 17 00:00:00 2001 From: "Alex I. Mandel" Date: Fri, 25 Sep 2026 09:40:55 -0700 Subject: [PATCH 5/5] typo Co-authored-by: Emmanuel Mathot --- decision-framework/index.qmd | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/decision-framework/index.qmd b/decision-framework/index.qmd index ae77ad2..8092aff 100644 --- a/decision-framework/index.qmd +++ b/decision-framework/index.qmd @@ -210,7 +210,7 @@ Every candidate format has a knob that sets how bytes are grouped into data chun Tune the lever to your [object store's](../glossary.qmd#object-storage-cloud-storage) **object-size policy** (S3, Azure Blob Storage, Google Cloud Storage, or an on-premises equivalent). Object stores degrade when they must index very many tiny objects, so most reward a larger average object. -In general, when it comes to reading from object storage, objects can be too large (readers have to download and decompress more data than they need) or too small (readers make many small requests and pay the network latency for each). See also: [the datacube guid on tiny chunks](https://developmentseed.org/datacube-guide/latest/worst-practices/tiny-chunks.html) and [massive chunks](https://developmentseed.org/datacube-guide/latest/worst-practices/massive-chunks.html). +In general, when it comes to reading from object storage, objects can be too large (readers have to download and decompress more data than they need) or too small (readers make many small requests and pay the network latency for each). See also: [the datacube guide on tiny chunks](https://developmentseed.org/datacube-guide/latest/worst-practices/tiny-chunks.html) and [massive chunks](https://developmentseed.org/datacube-guide/latest/worst-practices/massive-chunks.html). One example: the [CNES Datalake](https://cnes.fr/en/projects/centre-de-calcul) sorts data into [tiers](../glossary.qmd#tiered-storage) by mean object size, with about 32 MB or more for warm disk and about 100 MB or more for cold tape. This serves the general point: find out whether your storage has an object-size policy, and adjust your groupings accordingly.