diff --git a/.gitignore b/.gitignore index 66f46915..0f3b41e4 100644 --- a/.gitignore +++ b/.gitignore @@ -15,3 +15,5 @@ __pycache__/ .ipynb_checkpoints /.luarc.json + +**/*.quarto_ipynb diff --git a/CITATION.cff b/CITATION.cff index a8df392f..8de5cd6d 100644 --- a/CITATION.cff +++ b/CITATION.cff @@ -17,6 +17,8 @@ authors: family-names: "Barron" - given-names: "Zachary" family-names: "Deziel" +- given-names: "Emmanuel" + family-names: "Mathot" - given-names: "Max" family-names: "Jones" - given-names: "Brianna" diff --git a/_quarto.yml b/_quarto.yml index 81eb6654..fa7a485d 100644 --- a/_quarto.yml +++ b/_quarto.yml @@ -31,6 +31,15 @@ website: text: Welcome - href: overview.qmd text: Overview Slides + # NOTE for @abarciauskas-bgse: proposed nav placement only. + # Placed before "Formats" so a reader can pick a format family before + # diving into a format page. Final position depends on the Track-3 site + # redesign - please confirm or move (and rename the path if needed). + - section: Choosing a Format + href: decision-framework/index.qmd + contents: + - decision-framework/index.qmd + - decision-framework/case-studies.qmd - section: Formats contents: - section: Cloud Optimized GeoTIFFs (COG) @@ -89,7 +98,7 @@ format: html: include-in-header: plausible.html theme: - light: [cosmo, theme.scss] # from https://github.com/sta210-s22/website/blob/main/_quarto.yml + light: cosmo # from https://github.com/sta210-s22/website/blob/main/_quarto.yml dark: [cosmo, theme-dark.scss] code-copy: true code-overflow: wrap diff --git a/_thankyous.qmd b/_thankyous.qmd index c2b17772..47dce6f1 100644 --- a/_thankyous.qmd +++ b/_thankyous.qmd @@ -10,4 +10,6 @@ This guide has been made possible through the support of: [![](./images/thankyous-devseed-logo.png){fig-alt="Development Seed Logo" fig-align="left" width=40%}](https://developmentseed.org) [![](./images/thankyous-cng-logo.png){fig-alt="Cloud-Native Geospatial Forum" fig-align="left" width=40%}](https://cloudnativegeo.org) -::: \ No newline at end of file +::: + +The [Format Selection Decision Framework](./decision-framework/index.qmd) and its case studies were contributed from a study for the [Centre National d'Études Spatiales (CNES)](https://cnes.fr/), and use public [Copernicus](https://www.copernicus.eu/) and [NASA](https://www.earthdata.nasa.gov/) mission data. \ No newline at end of file diff --git a/decision-framework/case-studies.qmd b/decision-framework/case-studies.qmd new file mode 100644 index 00000000..20919da2 --- /dev/null +++ b/decision-framework/case-studies.qmd @@ -0,0 +1,227 @@ +--- +title: "Decision Framework Case Studies" +subtitle: "Six products run through the framework, five of them backed by measurements" +--- + +Each case study below runs a real product through the [decision framework](index.qmd). It shows where the product lands and why. Five of the six are backed by **measured benchmark results**. + +If a term here is unfamiliar, the framework page defines the framework's own vocabulary in [Key terms](index.qmd#key-terms), and the guide's [glossary](../glossary.qmd) covers general cloud-native terms. + +Each case study includes: + +1. The characteristics of the product, and what the data is for. +2. **The path through the framework**: the [control fork](index.qmd#control), the [topology question](index.qmd#topology), then the four refinements: [intent](index.qmd#intent), the [object-size and grouping lever](index.qmd#grouping-lever), [ecosystem maturity](index.qmd#maturity), and [interpretation encoding](index.qmd#interpretation-encoding). +3. **What the benchmark measurements showed**, for each candidate format. +4. The decision outcome. + +The examples use public Copernicus and NASA missions. The [CNES Datalake](https://cnes.fr/en/projects/centre-de-calcul) appears as one worked example of a [tiered object store](../glossary.qmd#tiered-storage). Its two **advisory** floors appear throughout the cases below: about 32 MB mean object size for the **warm tier** on disk, and about 100 MB for the **cold tier** on tape. Nothing enforces these floors. They keep the object count at a level that the shared index of the service can manage. It is not the only kind of object store, and its numbers are not a general rule. + +::: {.callout-note} +## Cloud-optimized != smaller storage size + +**A cloud-optimized format is not necessarily smaller than a non-cloud-optimized version.** + +Cloud-optimized formats provide partial access via byte-range indices into the data, accessible via addressable metadata. + +**The extra bytes are the feature.** Overviews, levels of detail, and spatial indexes are structures the source file did not carry at all. They add volume, and they are exactly what buys fast display and cheap subsetting. The COG in case 1 is about 30% larger than the delivered archive mostly because it carries overviews, and the row-group index in case 4 is the whole point of the conversion. + +**Where the overviews live is the real trade-off.** Overview depth costs a COG only bytes, because its overviews sit inside the same file. It costs a Zarr store objects, because each pyramid level is a separate array with its own objects. Cases 1 and 2 measure this directly, and it is the most important result in this set. +::: + +::: {.callout-note} +The numbers below come from the final benchmark campaign. Each figure is measured on a representative unit: an area block of 16 to 20 products for Sentinel-1 and Sentinel-2, and 2 to 12 granules or passes for the SWOT products. Every size comparison is like-for-like on content, and any content subset is flagged as one. The compression ratio, the codec contrast, and the tier classification are per-unit properties, so they carry to a full collection. The object count and the total volume scale with the number of units. + +They were produced with [`cng-formats-benchmark`](https://github.com/developmentseed/cng-formats-benchmark), an open-source harness that measures object-size distribution, write, read, and display per format. You can describe your own datasets in its config files and reproduce this shape of comparison on your own data. See [Optional: make it quantitative](index.qmd#optional-make-it-quantitative). +::: + +## 1. Per-scene optical raster: COG vs GeoZarr for visualization and distribution + +**Product:** Sentinel-2 L2A. **Source format:** geoTiff in a `.zip`. + +- **Existing data?** Yes. +- **[Can you reprocess?](index.qmd#control)** Yes, the producer operates the processing chain, so the format is a free choice. +- **[What shape are the data?](index.qmd#topology)** 2D raster per scene, which opens `{COG, GeoZarr}`. +- **[Intent](index.qmd#intent).** Distribution and visualization, both served per scene. That favors COG; time-series analysis would favor a GeoZarr cube instead. +- **[Grouping lever](index.qmd#grouping-lever).** COG internal tiling, or a scene-size Zarr shard. + +Both intents are served per scene, which is why this case does not need a cube at all. + +**Measured, 16-product area block.** Three arms carried the same pixels. COG came to 25478 MiB with ZSTD, in 448 objects at a mean object size of 57 MiB, which clears the [warm tier](../glossary.qmd#tiered-storage). A flat Zarr store with no overview pyramid came to 19432 MiB, about 24 % smaller. A pyramid-complete GeoZarr came to 26026 MiB, about 2 % *larger* than the COG, in 2510 objects at a mean of 10.4 MiB, which clears no tier at all. + +**The codec is not the driver. The pyramid is.** Swapping COG's own codec from ZSTD to Deflate changes its total by 2 %, from 25478 MiB to 25998 MiB. The 24 % that the flat Zarr store saves is the pyramid that it does not carry. Add that pyramid, and the saving disappears. + +**The pyramid buys display speed and costs tier fit.** With the pyramid, and with a reader that selects the matching multiscale level, zoomed-out display runs about 5x faster than the same store without a pyramid: 0.198 s against 0.98 s. The price is the object count, 5x more than that flat store, which takes this product below every tier. COG pays neither price, because its overviews sit inside the one file. + +**Against the delivered `.zip`, per scene.** The COG is 2625.7 MiB against a 2023.7 MiB source, so about 30 % larger. The overviews account for most of that. + +**[Interpretation encoding](index.qmd#interpretation-encoding) is a real differentiator here.** The source reflectance is stored as scaled integers, and the nodata fill is left undeclared. COG keeps the scale, the offset, and the nodata value in locations that the format does not standardize, such as a GDAL metadata tag. Readers do not apply them consistently. Zarr v3 can carry all three as first-class array properties, using the `scale_offset` codec and a declared fill value, so every reader recovers the same physical value. + +**Where it lands.** COG per scene for distribution and visualization. + +For temporal analysis GeoZarr is the right target, but **the GeoZarr store measured here does not yet deliver it.** What was written is a per-scene, two-dimensional store, so there is no time dimension to exploit. It proves the container and the encoding, and it carries the scale, offset, and nodata correctly. A time-series query against it still has to open one store for each date, which is the same fan-out that the COG layout has. + +A pyramid-less store measures smaller, as the numbers above show. Do not read that as the efficient option. Such a store is valid GeoZarr, because multiscales are optional in the convention, but it cannot serve a web map at speed. + +The temporal benefit appears only once many scenes are concatenated along time into a single cube. That is the regime measured in [case 2](#case-time-cube), and it is a collection-level layout decision, not something a scene-size store can give you. Read the per-scene store as a building block and the cube as the product. + +The related masking product is written as many small GeoTIFFs. Their mean size is far below the warm-tier floor. This is the small-object problem that the [grouping lever](index.qmd#grouping-lever) describes, and the producer must correct it at the source. + +Sentinel-2 also has a second multiscale dimension worth naming: its bands are natively captured at 10 m, 20 m, and 60 m, not derived from each other by downsampling. See [Multiscale and native mixed resolution](index.qmd#multiscale-and-native-mixed-resolution) for how one aligned pyramid can host all three without picking a single working resolution. + +## 2. Raster time-cube: GeoZarr, with COG as the alternative {#case-time-cube} + +**Product:** Sentinel-1 RTC. **Source format:** geoTiff in a `.zip`. + +- **Existing data?** Yes. +- **[Can you reprocess?](index.qmd#control)** Yes, so the format is a free choice. +- **[What shape are the data?](index.qmd#topology)** 2D raster over time, which opens `{COG, GeoZarr}`. +- **[Intent](index.qmd#intent).** Advanced analysis and visualization lead, with distribution close behind but weighted a little lower. +- **[Grouping lever](index.qmd#grouping-lever).** Zarr sharding, sized to the object store's preference for large objects. + +The analysis half is what separates this case from case 1: it is asked over time, not per scene. A year of RTC over a region is naturally a `time × y × x × pol` cube, and on a Zarr cube a time-series slice or an area-of-interest subset is one range-addressable query. Across thousands of per-timestamp COGs the same request becomes a large fan-out. Sharding reconciles fine-grained chunk access with the object store's preference for large objects. + +**Measured, 20-product area block.** COG came to 16449 MiB with ZSTD, in 40 objects at a mean of 411 MiB, which clears the [cold tier](../glossary.qmd#tiered-storage). A flat Zarr store with no overview pyramid came to 12344 MiB in 80 objects, about 25 % smaller, and it also clears the cold tier. A pyramid-complete GeoZarr came to 15867 MiB, about 3.5 % *smaller* than the COG, but in 480 objects at a mean of 33 MiB, which drops it from the cold tier to the warm tier. + +**The codec is not the story here either.** Deflate against ZSTD changes the COG total by 0.3 %, from 16498 MiB to 16449 MiB. Sentinel-1 backscatter carries speckle, and speckle resists general-purpose compressors whatever the algorithm. + +**The pyramid buys display speed and costs tier fit.** With the pyramid, and with a reader that selects the matching multiscale level, zoomed-out display runs about 10x faster than the same store without a pyramid: 0.171 s against 1.82 s. The speed also holds across zoom levels instead of collapsing at low resolution. The price is 6x the object count of that flat store, and one tier level. + +**Against the delivered `.zip`, per scene.** The COG is 728 MiB against a 496 MiB source, so about 47 % larger. Cloud-optimized costs storage here, and it buys the range access that the opaque zip cannot give at all. + +::: {.callout-important} +**The cube itself is not measured.** Every GeoZarr figure above describes a per-scene, two-dimensional store. That is the shape in which Zarr has least to offer. A true time cube, one tile across many dates with the chunks laid over `time`, `y`, and `x`, is the shape that this recommendation is actually about, and this study did not test it. Nothing blocks the test technically. Read the case as an argument for the cube, and the numbers as evidence about the per-scene shape only. +::: + +**Where it lands.** COG for the serving path, because it keeps both the tier and the display speed. GeoZarr for the analysis path, once the cube is built and measured. + +## 3. Point cloud: netCDF to COPC + +**Product:** SWOT L2_HR Pixel Cloud. **Source format:** netCDF. + +- **Existing data?** Yes. +- **[Can you reprocess?](index.qmd#control)** Yes, so the format is a free choice. +- **[What shape are the data?](index.qmd#topology)** Point cloud, which opens `{COPC, GeoParquet}`. +- **[Intent](index.qmd#intent).** Distribution, visualization, and data fusion. +- **[Grouping lever](index.qmd#grouping-lever).** COPC octree nodes; each granule already clears the tier on its own. + +The fusion intent matters most here: these points get combined with other water products, so a reader has to recover the same physical value per point and subset the same area cheaply. That is what makes the source's missing spatial index a real cost rather than an inconvenience. COPC is a clustered octree inside a LAZ file; the netCDF source has no spatial index and no level of detail, so reading a small geographic subset means scanning the whole granule. + +**Measured, content-complete, six granules.** The COPC came to 3518.6 MiB against 3621.4 MiB of source netCDF. That is size parity: about 3 % smaller in total, and between 6 % smaller and 5 % larger for each granule. LASzip does out-compress the source, at 2.72x to 3.45x against 2.56x, but not by enough to shrink the product. Each granule clears both tiers on its own, at a mean object size of 586 MiB. + +The real gain is what the netCDF lacks: + +- A **spatial index in three dimensions**. The octree subdivides in x, y *and* z, so a query can constrain a bounding box **and** an elevation range, reading only the overlapping nodes. That matters here because water surface elevation is the measurement, not merely a coordinate. +- A **level of detail**. The octree holds 9,925 to 17,635 nodes for each granule, at a depth of 9 to 10. This is the point-cloud analogue of COG overviews. + +**The index has a price.** An octree-node spatial query costs 3.3 s to 3.8 s on these granules, and about 7 s at four-way concurrency. That is the cost of a tree deep enough to answer the query correctly. + +**Geometry precision decides whether the octree is usable.** PIXC stores longitude and latitude in degrees, and height in metres. An octree cube sized on those raw extents splits on height instead of on the footprint, and the single LAS scale that follows quantizes longitude and latitude to about a metre. The measured build avoids both problems: it reprojects to a local UTM zone before it builds the octree, it writes that CRS into the file, and it thins each coarse level into a real overview. + +::: {.callout-caution} +An earlier measurement on this product looked like a 97 % size saving. It had dropped most of the per-point variables. Size comparisons must be like for like on content. +::: + +**Where it lands.** COPC. The cloud-optimized value is partial access and level of detail at size parity, not a size saving. GeoParquet is the alternative where columnar analytics dominate, though 3D pruning there is optional: the [spec](https://geoparquet.org/releases/v1.1.0/) allows `zmin` and `zmax` in the bbox covering, but the writer has to populate them and the engine has to use them. + +## 4. Vector and tabular: shapefile to GeoParquet + +**Product:** SWOT L2_HR Lake Single Pass. **Source format:** shapefile. + +One pass is one overflight of the satellite. The mission delivers one shapefile for each pass. Each shapefile reports the whole prior lake database, and not only the lakes that the pass observed. About 78 % of the rows therefore describe a lake with no observation, and they carry a NULL geometry. The GeoParquet output keeps the same rows. + +- **Existing data?** Yes. +- **[Can you reprocess?](index.qmd#control)** Yes, so the format is a free choice. +- **[What shape are the data?](index.qmd#topology)** Vector and tabular, which opens `{GeoParquet, FlatGeobuf}`. +- **[Intent](index.qmd#intent).** Distribution, download, and data fusion. Visualization is *absent* from this mix. +- **[Grouping lever](index.qmd#grouping-lever).** Row groups, but only after aggregating across passes; a single pass is below the warm-tier floor. + +The intent mix makes this case a genuine two-way contest, though not in the way it first appears: + +- **Download favors GeoParquet, not FlatGeobuf.** FlatGeobuf has [no internal compression](../flatgeobuf/intro.qmd#no-internal-compression), and it cannot simply be gzipped, because its spatial index addresses byte ranges in the *uncompressed* file. GeoParquet [compresses per column by default](../geoparquet/index.qmd#internal-compression-by-default), so within this candidate set it is the smaller transfer. +- **Streaming is a visualization property, not a download one.** FlatGeobuf [streams](../flatgeobuf/intro.qmd#streaming-features-is-supported), so features render progressively as they arrive. Valuable, but visualization is absent from this mix. +- **FlatGeobuf wins on whole-row spatial subsetting.** It is row-oriented with a Hilbert R-tree, so every attribute of the features in a bounding box is one contiguous read, where a [column-oriented](../geoparquet/index.qmd#column-oriented) store has to touch every column. The measurements below put this at about 10x faster. It is a real access pattern, but it is not one that this intent mix asks for. + +At volume the scale question would matter too, because GeoParquet shards across [multiple files](../geoparquet/index.qmd#multi-file-support) while FlatGeobuf is a single file with a single index. The SWOT L2_HR passes are tiny, so the scale question does not matter here. + +So GeoParquet wins on download *and* fusion. The product is fixed-schema tabular, with a lake id, a reach id, water-surface elevation, area and more, which is the native fit for its columns and row groups. A sparse Zarr store would add complexity with no gain. + +**Measured, two passes.** GeoParquet came to 17.8 MiB per pass, so 35.6 MiB for the two passes, against 31.1 MiB for the same two passes as a zipped shapefile. That is about 14 % **larger**, and the ratio is the same per pass (17.8 against 15.6 MiB). + +Note that this compares GeoParquet against the **source**, not against FlatGeobuf. + +**Measured, three passes, both candidates.** GeoParquet with ZSTD came to 47.9 MB. FlatGeobuf with its R-tree index came to 75.1 MB, about 57 % larger. The R-tree itself is only about 2 % of the object, so the difference is the compression that FlatGeobuf cannot do. A bounding-box query, however, runs about 10x faster on FlatGeobuf: 0.049 s against 0.522 s warm. + +**FlatGeobuf could not hold the product as delivered.** A FlatGeobuf R-tree cannot index a NULL geometry, and 78 % of the rows carry one. The format could not take the product until those rows had a geometry. The run synthesized 20,689 geometries on the largest pass, from the reference point that each unobserved lake already carries in its attributes, and it recorded them as synthesized. This is a property of the shape of this product, not a fault in FlatGeobuf. It is also a good example of the data itself disqualifying a candidate. + +**The codec question is closed.** Snappy is 12 % larger than ZSTD on this product, so ZSTD was already the better default. The 14 % against the zipped shapefile is a property of the format pair, not of a weak setting. + +Each pass sits below the warm-tier floor. The lever is therefore **aggregation across passes**: concatenate the rows from several passes into one file, then size the row groups. A one-for-one transcode of each pass would leave the object-size problem in place. The gain over the shapefile is the bounding-box row-group index, not the size. + +**Where it lands.** GeoParquet, with pass-level aggregation. It wins on size, on download, and on fusion. FlatGeobuf is the measured alternative where whole-row spatial subsetting dominates, and it is about 10x faster for that query, but it costs 57 % more storage and it needs the NULL geometries fixed first. + +## 5. Small N-D grid: the aggregation regime + +**Product:** SWOT L2_HR Raster. **Source format:** geoTiff. + +- **Existing data?** Yes. +- **[Can you reprocess?](index.qmd#control)** Yes, so the format is a free choice. +- **[What shape are the data?](index.qmd#topology)** N-D grid, which opens `{GeoZarr, COG}`. +- **[Intent](index.qmd#intent).** Distribution and download, with data fusion against the other SWOT products, and some visualization. This mix is provisional and still to be confirmed with the product owner. +- **[Grouping lever](index.qmd#grouping-lever).** Aggregation *above* the product, not sharding within it. A 100 m granule covers a small tile, about 160 km across, so it is far below the warm-tier floor. + +**Measured, 12 granules, all 38 CF variables.** One object for each variable gives 456 COG objects, at a mean of 2.55 MiB and 1107.8 MiB in total. The GeoZarr equivalent gives 2640 objects, at a mean of 471 KiB and 1214.8 MiB in total. Neither clears any tier. + +**The conversion creates a small-object problem that the source does not have.** The source is one netCDF for each granule. One object for each of its 38 variables multiplies the object count by 38, and the smallest COG object is 19 KB. The framework calls this the small-object anti-pattern, and here the conversion introduces it. + +**The GeoZarr side is worse for a configuration reason, not a format reason.** The shard was set to 2048 px against a grid of about 1457 px. The shard was therefore wider than the whole array, it clamped to a single chunk, and the sharding lever grouped nothing. One store for each variable also repeats the `x` and `y` coordinate arrays and the pyramid metadata 38 times. A smaller chunk shape was not tried. + +::: {.callout-caution} +**The content-subset trap, in numbers.** A COG of the water-surface-elevation variable alone is 4.3 MiB, against about 67 MiB for the whole granule. The apparent 92 % saving is dropped content, not compression. Compare like for like on content, as in case 3. +::: + +The natural aggregation unit is a spatial mosaic, for each grid zone and each orbit cycle. That turns a wall of tiny granules into one object that a reader can address with range requests. + +::: {.callout-warning} +**Aggregation is sized, not measured.** This study wrote no consolidated object and read none. The decisive question stays open: does a large aggregate stay range-addressable? A bounding-box read on the aggregate must cost about what the same read costs on the small units. Treat every aggregation figure in these cases as arithmetic, never as a measurement. +::: + +**Where it lands.** COG for the small raster, plus aggregation across variables and granules. GeoZarr is the alternative to compare once the shard is sized against this grid. + +This case is the counterpart to cases 1 and 2. There, one large band already clears the tier, so you group *within* a product. Here the products are small, so you group *above* a single product. + +## 6. Existing archive to a virtual store + +**Product:** a large immutable archive in a chunked array format, for example HDF5 or netCDF. **Source format:** whatever was delivered. + +- **Existing data?** Yes. +- **[Can you reprocess?](index.qmd#control)** No. +- **Can you repackage the delivered files?** No. +- **[Virtualizable?](index.qmd#control)** Yes, *if* the source is a chunked array format. Kerchunk and VirtualiZarr cover netCDF4, HDF5, GRIB2, TIFF and FITS. Most vector formats have no equivalent, so this branch is effectively raster and N-D only. +- **[Grouping lever](index.qmd#grouping-lever).** None available. You inherit the source's chunking exactly. + +This case has no benchmark. It illustrates the "existing data, cannot reprocess" branch of the control fork, and it is the only case that does not reach the topology question at all. + +An immutable archive, for example HDF5 or netCDF that you cannot reprocess, can be surfaced as a virtual Zarr store with [VirtualiZarr, Kerchunk, or Icechunk](../zarr/virtual-zarr.qmd). This gives range-based access without rewriting the data. + +The result is more limited than a native store. Chunk sizes are whatever the source chose, you cannot add overviews, and you inherit the source's codec. It is still much better than whole-file download, and it is often the only option available. + +**"Cannot reprocess" usually means "too big".** This branch is normally reached for reasons of scale rather than permission: the archive is large enough that duplicating it, or spending the compute to regenerate it, costs more than the access problem is worth. That is why it is mostly an array and raster story. A vector dataset is usually small enough that [repackaging](index.qmd#control) it outright is cheaper than indexing around it, so vector products rarely land here. + +This case is also where the ["not yet possible"](index.qmd#not-yet-possible) framing matters. If the source cannot even be virtualized, name what is missing rather than declaring the case impossible. + +## Summary + +| Case | Topology | Lands on | Size against source | Main gain | +| --- | --- | --- | --- | --- | +| 1. Optical scene | 2D raster | COG per scene | COG +30 % | Display speed at full tier fit | +| 2. SAR time-cube | Raster over time | COG to serve, GeoZarr to analyse | COG +47 % | One query instead of a fan-out | +| 3. Pixel cloud | Point cloud | COPC | parity, −3 % | 3D spatial index, level of detail | +| 4. Lake vector | Vector, tabular | GeoParquet, aggregated | +14 % | Row-group bbox index | +| 5. Small raster grid | N-D grid | COG, plus aggregation | not computed | Fewer, larger objects | +| 6. Frozen archive | Chunked arrays only | Virtual Zarr | No rewrite | Range access with no reprocessing | + +Read the size column together with the note at the top of this page. **No case came out smaller than its source.** Case 3 is the closest, and it lands at parity. In cases 1 and 2, the choice between the two candidates is not about size at all: both carry the same pixels for about the same bytes once both carry overviews. The choice is about where the overviews live, and what that costs in objects. + +## Acknowledgements + +These case studies come from a study for the [Centre National d'Études Spatiales (CNES)](https://cnes.fr/), carried out by [Development Seed](https://developmentseed.org/). They use public Copernicus and NASA mission data. The measurements were produced with [`cng-formats-benchmark`](https://github.com/developmentseed/cng-formats-benchmark), which is maintained separately from this guide so that its methodological choices stay debatable in their own right. See [Our Supporters](../contributing.qmd#thank-you-to-our-supporters). diff --git a/decision-framework/index.qmd b/decision-framework/index.qmd new file mode 100644 index 00000000..8092aff9 --- /dev/null +++ b/decision-framework/index.qmd @@ -0,0 +1,361 @@ +--- +title: "Format Selection Decision Framework" +subtitle: "How to go from data you must publish to a short list of cloud-optimized format candidates" +--- + +## What this framework is + +This framework helps you go from *"I have geospatial data to publish or migrate"* to a **short list of cloud-optimized format candidates**, and to the **layout choices** that make each candidate work. + +It is a guide to help make informed decisions, and does not prescribe a single correct format. + +- **What you control** comes first: whether the data already exist, whether you may reprocess them, and whether a mandate fixes an archival format. This can settle the answer before format even comes up. +- The **shape of your data** narrows the set of candidate formats. +- The **intended use** and a few practical levers rank the remaining options. +- **You make the final call.** + +The framework is data- and cloud-provider-neutral. Within data and cloud-provider contexts, a rule may help make a decision. For example, an object-size policy or a benchmark result may influence the choice. These cases are given as worked examples, not rules. + +This is a first, deliberately simple version. It aligns on the questions to ask and the order to ask them in. Follow the links to a format page when you need more detail. + +::: {.callout-note} +## What this framework is not + +- It is not a compliance checklist. No format is disqualified by a single answer. +- It does not prescribe a catalog or a metadata standard. See [Metadata and cataloging](#metadata-and-cataloging). +- It does not replace a benchmark. When two candidates are close, measure them. +::: + +## Key terms + +The framework uses a few terms in a specific way. Each links to the section that develops it. + +| Term | What it means here | +| --- | --- | +| [**Access pattern**](../glossary.qmd#access-pattern) | What subsets readers will request: a time series at one point, a spatial window at one date, a full scan. Intent implies an access pattern, and your layout either serves it or fights it. Defined in the guide's glossary; the layout consequences are in [the grouping lever](#grouping-lever). | +| [**Archival-format mandate**](#control) | A requirement to maintain a specific archival format for preservation, such as HDF5, netCDF or GRIB, whatever cloud-optimized format you publish alongside it. | +| [**Control**](#control) | Whether the choice is yours to make at all: do the data already exist, and may you reprocess them, technically and legally. This is Stage 1 of the framework, and it can end in "not yet possible". | +| [**Ecosystem maturity**](#maturity) | How settled a format is, on an L1 to L5 scale from experimental to de-facto standard (see below). It answers "safe default, or bet?". | +| [**Grouping lever**](#grouping-lever) | The knobs each format gives you for grouping bytes into data chunks and objects in cloud storage: COG internal tiling, Zarr v3 chunking and sharding, COPC octree nodes, GeoParquet row groups. | +| [**Intent**](#intent) | What the data are for: archive and download, processing, visualization, distribution, or data fusion. Most products serve several intents at once. | +| [**Interpretation encoding**](#interpretation-encoding) | Whether the format carries units, CRS, nodata and any scale and offset transform in-format, so every reader recovers the same physical value, or leans on a side catalog. | +| [**Object size**](#grouping-lever) | The size of the objects your layout actually writes to storage. Object stores reward a larger average object and many apply a minimum size policy. | +| [**Refinement**](#refinements) | A criterion applied *inside* a format family to rank its members. There are four: intent, the grouping lever, ecosystem maturity, and interpretation encoding. | +| [**Topology**](#topology) | The shape of the data: 2D raster, N-D cubes, point cloud, or vector and tabular. Answering *what shape are the data?* selects a format family, not a format. | +: {tbl-colwidths="[15,85]"} + +## Preconditions + +This guide already defines what makes a format cloud-optimized. The Welcome page states it as a four-part [cloud-optimized format pattern](../index.qmd#the-opportunity), and the glossary defines [cloud-optimized](../glossary.qmd#cloud-optimized) as the ability to read a meaningful *part* of a file without downloading *all* of it. + +The framework starts from the definition on the Welcome page. Two requirements are worth explaining in more detail, because they decide whether a candidate belongs in the framework at all: + +**P1. The format must support partial and parallel reads over HTTP.** [Random access](../glossary.qmd#random-access) through [HTTP range requests](../glossary.qmd#http-range-request) is what makes a format usable from [object storage](../glossary.qmd#object-storage-cloud-storage). It depends on the rest of the pattern: [metadata](../glossary.qmd#metadata) in a consistent place that a reader can load in a single read, and then addressable [chunks](../glossary.qmd#chunk), internal tiling, or both. A reader that cannot locate the bytes it needs from the metadata alone cannot fetch only those bytes. + +**P2. No opaque bundles.** An [archive format](../glossary.qmd#archive-format) is not disqualified just for being an archive. [PMTiles](../pmtiles/intro.qmd) is an archive and it is cloud-optimized, because its internal directory makes each tile addressable. What disqualifies a candidate is an archive with no addressable internal layout, or a layer of [external compression](../glossary.qmd#external-compression) wrapped around data that was already cloud-optimized. A zipped Zarr store, a LAZ file without [COPC metadata](https://copc.io/), and a single Parquet file with no row groups all fail this way: the client must fetch the whole object before it can use any part of it. + +## The two stages + +The framework has **two stages**: + +**Stage 1 — control: do you have a choice?** Do the data already exist? If they +exist, are you able (technically, legally) to reprocess them? This stage can end +the process. If you cannot reprocess the data and cannot virtualize them, there +is no format decision to make yet. + +**Stage 2 — topology and refinement: which format?** You reach this stage only +when Stage 1 leaves you free to choose. Here the shape of the data selects a +**format family**: the small group of cloud-optimized formats built for that +shape. The refinements below rank the options within a format family. + +Four refinements then rank the candidates: **intent**, the **object-size and +grouping lever**, **ecosystem maturity**, and **interpretation encoding**. The +last one deserves a close look, because the formats differ on it more than +people expect. + +```{mermaid} +flowchart TD + A([Data to publish or migrate]) --> B{Existing data?} + + B -- No --> C{Archival-format
mandate?} + C -- Yes --> C1[Optimize the mandated format
+ publish a cloud-optimized copy] + C -- No --> T{{What shape are the data?}} + + B -- Yes --> D{Can you
reprocess?} + D -- Yes --> T + D -- No --> P{Can you repackage
the delivered files?} + P -- Yes --> P1[Format free,
content fixed as delivered] + P1 --> T + P -- No --> E{Virtualizable?} + E -- Yes --> E1[VirtualiZarr / Kerchunk / Icechunk
limited, better than nothing] + E -- No --> E2[Not yet possible
say what is missing] + + T --> T1(2D raster
COG, Zarr, PMTiles) + T --> T2(N-D cubes
GeoZarr, COG, CO-HDF5) + T --> T3(Point cloud
COPC, GeoParquet) + T --> T4(Vector / tabular
GeoParquet, FlatGeobuf, PMTiles) + + T1 --> R([Refine: intent, object-size lever,
maturity, interpretation]) + T2 --> R + T3 --> R + T4 --> R + + classDef entry fill:#e7eefb,stroke:#3f6db3,stroke-width:2px,color:#152744 + classDef ask fill:#fff6dc,stroke:#ad7c00,stroke-width:2px,color:#4a3600 + classDef hinge fill:#dfeffa,stroke:#2a7fb8,stroke-width:3px,color:#0e3247 + classDef family fill:#e4f4ea,stroke:#39875f,stroke-width:2px,color:#12301e + classDef partial fill:#fdeadb,stroke:#c06520,stroke-width:2px,color:#48260a + classDef refine fill:#ece6f7,stroke:#7355a6,stroke-width:2px,color:#291d44 + + class A entry + class B,C,D,E,P ask + class T hinge + class T1,T2,T3,T4 family + class C1,P1,E1,E2 partial + class R refine + +``` + +## The control fork {#control} + +**No existing data.** You are designing a new product, so you can choose freely. Go to the [topology question](#topology). + +**If an archival-format mandate applies,** it adds a second job, and it does not remove your freedom to choose how readers get access. Whatever the mandated format, make it as range-request-friendly as it can be. Where the mandated format is HDF5 or netCDF, this guide has [specific guidance](../cloud-optimized-netcdf4-hdf5/index.qmd) on consolidated metadata and chunk sizing. What is genuinely optional is what you do beyond the archival format. + +You can wrap the optimized archive in a [virtual Zarr](../zarr/virtual-zarr.qmd) layer: cheaper to maintain, but you inherit the archive's chunk shape, the same limit as [existing data you cannot reprocess](#control) below. The stronger, recommended option is to also publish a cloud-optimized copy for readers, since a cloud-optimized copy is an opportunity to provide access beyond the archive's native chunking. + +**Existing data you can reprocess.** You control the production chain, so you can also choose freely. Go to the [topology question](#topology). + +**Existing data you cannot reprocess, but can rewrite.** You may still be allowed to repackage the delivered files: a plain TIFF becomes a COG, a LAZ file becomes a COPC, a shapefile becomes GeoParquet. The format is a free choice, but the content is fixed at whatever was delivered. Go to the [topology question](#topology). + +**Existing data you cannot reprocess.** You can often still publish a virtual view over the bytes you have. The [Virtual Zarr](../zarr/virtual-zarr.qmd) approach is to use a library (such as VirtualiZarr or Kerchunk) to build an index that points into the original files. That index is then stored in formats such as Icechunk, kerchunk json, or kerchunk parquet. This gives range-based access without rewriting the data. The result is more limited than a native store, but it is much better than nothing. + +**Neither.** Some sources cannot be repackaged or virtualized today. Examples are HDF4, encrypted files, and a single large blob with no internal structure, whether or not it's compressed. See [Not yet possible](#not-yet-possible). + +### How to tell which branch you are on + +Whether you can reprocess or virtualize usually depends on the data and data project context. + +**You can reprocess** if you have the processor, the inputs, and the budget to re-run +the production chain. Then format and layout are both free choices: go to +[topology](#topology). + +**You can repackage** if you cannot re-run the chain but you may rewrite the delivered +files. A plain TIFF, JPEG2000, or MrSID image becomes a COG, a LAS or LAZ file becomes +a COPC, a shapefile or CSV becomes GeoParquet. The rewrite can be drastic and still +count as repackaging, not reprocessing: an HDF5 granule can become a COG, and many +single-scene COGs can become one Zarr cube, as long as the underlying values are not +recomputed. Repackaging is a byte-level rewrite rather than a science reprocessing, +so it is often permitted where reprocessing is not. Format is a free choice; the +content is fixed at whatever was delivered. + +**You can virtualize** if the bytes you have can be indexed as they are. +[Virtual Zarr](../zarr/virtual-zarr.qmd) publishes that index without copying the data. +Three tests, in order: + +1. **Is the data chunked internally?** This is the hard requirement. A virtual store + cannot change the source's chunk structure, so a single contiguous block virtualizes + into one enormous chunk and buys you no useful partial access. +2. **Are those chunks a workable size?** You mostly inherit the source's chunk size and shape, so its chunking limits which access patterns stay cheap. There is little room to make a store built for one access pattern also serve a very different one well. +3. **Is there a parser for the format?** VirtualiZarr and Kerchunk cover the common + archival formats, including netCDF4, HDF5, GRIB2, TIFF and FITS. + +**Otherwise there is no path yet.** The usual blockers are HDF4, encrypted +files, a codec no parser can pass through, and a single blob with no internal +chunking, whether or not it's compressed. See [Not yet possible](#not-yet-possible). + +**Passing these tests means you *can*, not that you *should*.** A producer may be +technically and legally free to change format and still be held by what its users' +tools expect. See [Existing users and their tools](#existing-users). + + +## Topology: what shape are the data? {#topology} + +The shape of the data selects a **format family**: the set of format candidates the refinement process narrows down. [Intent](#intent) is what decides between the members. + +| Data shape | Format family | Where each member wins | +| --- | --- | --- | +| 2D raster, per scene or per tile | [COG](../cloud-optimized-geotiffs/intro.qmd), [Zarr](../zarr/intro.qmd), [PMTiles](../pmtiles/intro.qmd) | COG for distribution and visualization; Zarr if the product is really a cube; PMTiles for pre-rendered tile delivery | +| N-D cubes, for example `time × y × x × band` | [[Geo]Zarr](../zarr/intro.qmd), [COG](../cloud-optimized-geotiffs/intro.qmd), [Cloud-Optimized HDF/NetCDF](../cloud-optimized-netcdf4-hdf5/index.qmd) | GeoZarr for analysis across the cube; COG per time step if slices are read alone; cloud-optimized HDF/netCDF where an archival format is mandated or existing tooling must keep working | +| Point cloud | [COPC](../copc/index.qmd), [GeoParquet](../geoparquet/index.qmd) | COPC for spatial subsetting and level of detail; GeoParquet if columnar analytics dominate | +| Vector or tabular | [GeoParquet](../geoparquet/index.qmd), [FlatGeobuf](../flatgeobuf/intro.qmd), [PMTiles](../pmtiles/intro.qmd) | GeoParquet for joins and analytics; FlatGeobuf for streaming download; PMTiles for tiled web delivery | + + +The question is about topology, not about the format the data arrives in. A netCDF granule that holds a point cloud belongs on the point-cloud branch. See the [format table](../index.qmd#data-type-to-traditional-to-cloud-optimized-geospatial-file-format-table) for how traditional formats map to cloud-optimized ones. + +## Refinements {#refinements} + +These refinements help rank candidate formats within a family. + +### Intent {#intent} + +What are the data for? Archive and download, processing, visualization, distribution, or data fusion. Three points matter: + +- **Visualization is a first-class intent**, not a side effect. It has its own layout needs, mainly [overviews](../glossary.qmd#overviews) or a level of detail. +- **On-the-fly and batched processing pull the layout in opposite directions.** On-the-fly access rewards small chunks, fine overviews, and cache locality: a reader fetches a little and wants it back fast. Batched processing rewards large shards or row groups, so that a bulk scan becomes a few large contiguous reads that many workers can run in parallel, instead of thousands of small scattered ones. +- **Data fusion raises the stakes on encoding.** Combining products from different missions or processors means every input must be read back as the same physical value, and subsets from each must line up. Fusion is a form of processing, but it is worth naming separately: it weights [interpretation encoding](#interpretation-encoding) and consistent grids far higher than single-product analysis does, and it usually rewards a tabular or columnar target where joins are cheap. + +Most real datasets serve several intents at once. However, the intents often carry different weight in the decision process. The [case studies](case-studies.qmd) state an intent mix for each product, which is what decides the ranking inside a candidate set. + +### The object size and grouping lever {#grouping-lever} + +Every candidate format has a knob that sets how bytes are grouped into data chunks and chunks grouped into objects: + +| Format | Grouping lever | +| --- | --- | +| COG | internal tiling and overviews | +| Zarr v3 | chunks and shards ([chunk](../glossary.qmd#chunk) and [shard](../glossary.qmd#shard)) | +| COPC | octree nodes | +| GeoParquet | row groups, plus partitioning | + +Tune the lever to your [object store's](../glossary.qmd#object-storage-cloud-storage) **object-size policy** (S3, Azure Blob Storage, Google Cloud Storage, or an on-premises equivalent). Object stores degrade when they must index very many tiny objects, so most reward a larger average object. + +In general, when it comes to reading from object storage, objects can be too large (readers have to download and decompress more data than they need) or too small (readers make many small requests and pay the network latency for each). See also: [the datacube guide on tiny chunks](https://developmentseed.org/datacube-guide/latest/worst-practices/tiny-chunks.html) and [massive chunks](https://developmentseed.org/datacube-guide/latest/worst-practices/massive-chunks.html). + +One example: the [CNES Datalake](https://cnes.fr/en/projects/centre-de-calcul) sorts data into [tiers](../glossary.qmd#tiered-storage) by mean object size, with about 32 MB or more for warm disk and about 100 MB or more for cold tape. This serves the general point: find out whether your storage has an object-size policy, and adjust your groupings accordingly. + +Some providers publish a threshold, some only charge you as if your objects were larger, and some have no policy at all. + +**Size and shape are two different questions.** How *big* the objects are is a +question for your object store, answered above. How the bytes are arranged *inside* them +(chunk shape, and the order of the dimensions) is an **access-pattern** question, +and an object store provider has no opinion about the shape. A `time × y × x` cube chunked long in time +serves a time series at one point cheaply and a full-scene map expensively; chunked +wide in `y` and `x` it does the opposite. This guide works it through, with a +diagram, in [chunk shape vs chunk size](../cloud-optimized-netcdf4-hdf5/index.qmd#chunk-shape-vs-chunk-size). + +Shape rarely decides *which* format you pick, because every candidate in a family lets +you set the shape: a COG chooses its tile dimensions, Zarr its chunk shape and dimension +order, GeoParquet its row-group size and row ordering. What shape decides is how well +the format you picked performs, not which one you pick. And it is mostly settled back +at the [control](#control) question, because only reprocessing lets you choose it freely. + + +### Ecosystem maturity {#maturity} + +Ecosystem maturity may influence the final decision. You may need something production-grade that you can count on, or you may have the flexibility to be an early adopter. Maturity is what separates a safe default from a bet. + +To keep that judgement explicit rather than impressionistic, this framework assigns candidate formats on the following five-level scale: + +| Level | Meaning | +| --- | --- | +| **L5** | De-facto standard: universal tooling, no specification risk | +| **L4** | Established: released spec, broad independent tooling, in production use | +| **L3** | Stable: released spec, multiple implementations, growing tooling | +| **L2** | Emerging: spec stabilizing, reference implementations, early adopters | +| **L1** | Experimental: draft or in-flux spec, few implementations | + +Where the candidates sit today: + +| Format | Specification | Tooling | Level | +| --- | --- | --- | --- | +| [COG](../cloud-optimized-geotiffs/intro.qmd) | De-facto standard, on a GeoTIFF base | GDAL, rio-tiler, TiTiler, every GIS | **L5** | +| [GeoParquet](../geoparquet/index.qmd) | v1.x, stable, on an Apache Parquet base | GDAL, GeoPandas, DuckDB, Arrow | **L4** | +| [Zarr](../zarr/intro.qmd) | v2 and v3 stable | GDAL, zarr-python, zarrs | **L4** | +| [COPC](../copc/index.qmd) | v1.0, stable, on a LAZ base | PDAL, untwine, QGIS, copc.js | **L4** | +| [FlatGeobuf](../flatgeobuf/intro.qmd) | Stable, on a flatbuffers base | GDAL and OGR, QGIS, flatgeobuf.js | **L3** | +| [GeoZarr](../zarr/intro.qmd) | Draft, in an OGC Standards Working Group | Emerging, mostly xarray-based stacks | **L2** | + +**COG is the reference case** and the format the others are measured against: a de-facto standard for years, supported by every relevant library and GIS, with no open specification risk. + +**For GeoParquet and COPC, the risk is integration effort, not the specification.** Each inherits a mature base format and adds a spatial index on top. + +#### New Zarr Standards: Distinguishing Zarr v3 and GeoZarr + +This distinction is easy to miss, and it decides how much risk you actually take on. + +- **Zarr v3, the substrate of GeoZarr, is dependable.** The core specification was finalized in 2024. [Sharding](#grouping-lever), the feature that matters most for object storage, is stable in `zarr-python` 3.x and readable by GDAL and xarray. Judged on its own, the Zarr v3 core behaves like an L4. +- **GeoZarr, the convention layer, is still being ratified.** It is what makes a generic [Zarr](../glossary.qmd#zarr) store *geospatial*: CRS, dimension semantics, and multiscale pyramids. It does for Zarr roughly what the GeoTIFF tags do for a plain TIFF. It is a draft in an OGC Standards Working Group, on the standards track but not yet an adopted OGC standard, and the published draft can lag behind active convention work. Check [geozarr.org](https://geozarr.org/) for the current status before you rely on a specific version. + +So the L2 above belongs to the convention layer, not to Zarr. The leading reference implementations of GeoZarr today, such as ESA's EOPF Sentinel output, are demonstrators rather than production guarantees. + +#### Maturity is a warning, not a veto + +An L2 candidate can still be the right answer. Multidimensional cubes have no mature format candidate. A low level does not mean "choose something else". It means **plan for change**. Budget for convention churn, pin the versions you depend on, and expect visualization tooling in particular to improve as tooling catches up. + +Many factors influence what maturity level you can tolerate. A one-off internal product and a mission archive with a twenty-year retention commitment can reasonably land on different levels from the same set of candidate formats. + +### Interpretation encoding {#interpretation-encoding} + +Does the format carry the information a reader needs to recover the same physical value, or does it lean on a side catalog? Interpretation encoding includes units, the [CRS](../glossary.qmd#coordinate-reference-system-crs), the nodata fill value (see also [Virtualizarr's distinction of fill values](https://virtualizarr.readthedocs.io/en/latest/explanation/custom_parsers.html#fill-values)), and any scale and offset transform. + +Interpretation encoding separates the candidates more than people expect: + +- **Zarr v3 makes the information first-class.** With the `scale_offset` codec, a declared fill value, Zarr conventions for projection, CRS and multiscales, the transform is a first-class array property. Every reader recovers the same value. +- **COG, GeoParquet, and COPC standardize the CRS** but leave the value transform to side metadata. Readers honor side metadata unevenly. + +If your product stores scaled integers, or if an undeclared fill value would change a computed result, weigh this aspect heavily. + +## Cross-cutting considerations + +These considerations do not impact what format you choose, but they do affect how you implement that format. + +### Metadata and cataloging + +Metadata that a reader can fetch in a single request is part of [Precondition P1](#preconditions), not an extra. Every candidate the [topology question](#topology) opens has its own mechanism for it: Zarr consolidated metadata, COG headers, GeoParquet footers, and the COPC header. What this section adds is the level *above* a single file. + +At a scale of tens of thousands to millions of files, collection-level discovery becomes critical. [STAC](https://stacspec.org/) has an essential role there, and it is part of the cloud-native geospatial ecosystem that we recommend. It is **not a format and not a decision point in this framework**, so this page does not prescribe it. Choose a catalog approach that fits your organization. + +### Within-format depth + +Some choices sit below the format and are foundational: + +- Base codecs, such as [ZSTD](../glossary.qmd#zstd) and [Deflate](../glossary.qmd#deflate) +- Data types, and scale-offset encoding +- Sharding in Zarr v3 + +These are mature and in scope. The per-format pages cover them. More experimental optimizations, such as bit-rounding and advanced compression tricks, are out of scope at this time. + +Formats also differ in which data types they support, and in whether types can be mixed within one file, column, or schema. Those specifics live on each format's own page and are worth checking before committing to a candidate; they can rule one out as surely as topology or intent. + +### Multiscale and native mixed resolution + +Providing downsampled levels of original data in the source resolution is a common pattern for providing good performance, usually for visualization. A common example is COG overviews, which provide tiles at standard resolutions for mapping clients. A multiscale pyramid is a more general concept: a set of levels of detail, each with its own resolution, that can be used for visualization or analysis. + +Often these multiscale pyramids are generated at a uniform decimation so a reader can zoom out without reading and resampling full-resolution chunks. COG overviews, COPC's octree levels of detail, and most GeoZarr pyramids work this way, and it is already covered above under [ecosystem maturity](#maturity) and the [grouping lever](#grouping-lever). + +How these multiscale pyramids are generated can vary: Sometimes, data is provided at varying resolution where lower resolution data is not derived from the higher resolution data. Sentinel-2 is an example, worked through in [case 1](case-studies.qmd) of the case studies: Sentinel-2 collects data in 10m, 20m, and 60m bands from the same sensor. These bands can become a pyramid by aligning the decimation to each step (10m to 20m halves the resolution, 20m to 60m divides it by three), and the pyramid can continue past 60 m into ordinary overviews using the same mechanism. [Zarr multiscales convention](https://github.com/zarr-conventions/multiscales) allows for non-uniform scale factors. + +This still interacts with the [grouping lever](#grouping-lever), since each native resolution needs its own tiling or chunk shape, and with [interpretation encoding](#interpretation-encoding), since each band needs its own geotransform. What changes from the plain-overview case is that these become refinements within one multiscale structure. + +### Existing users and their tools {#existing-users} + +Being able to reprocess is not the same as being free to change format. If a long-standing user community reads your product with established tools, changing format breaks them, and that cost is real even when nothing technical or legal stops you. + +This resembles an [archival-format mandate](#control) but behaves differently: + +| | Archival-format mandate | User and tooling expectations | +| --- | --- | --- | +| Comes from | Institutional policy | Your consumer community | +| Negotiable | Rarely | Yes, with notice | +| Ends when | Never, you satisfy it indefinitely | You finish a migration you announced | + +In the short term the answer is usually the same: publish the cloud-optimized format alongside what your users already consume. The endpoint differs. A mandate you keep satisfying; an expectation you migrate out of, with a deprecation window and documentation. Weigh it together with [ecosystem maturity](#maturity), because the newer your target format, the more of your users' toolchain you are asking them to replace. + +### Not yet possible {#not-yet-possible} + +When a source cannot be repackaged or virtualized today, the honest answer is *"not yet implemented"*, not *"impossible"*. The blocker is usually a missing parser, an unsupported codec, or source files with no internal chunking at all. Every one of those can change. + +The precedent is HDF5 itself: once the standard example of a format that could not be read efficiently from object storage, until a change in write strategy and reader tooling made it viable, see [Cloud-Optimized HDF/NetCDF](../cloud-optimized-netcdf4-hdf5/index.qmd). The reader half is still moving: switching xarray's default cache for the h5netcdf engine improves dataset opening ([xarray #11216](https://github.com/pydata/xarray/pull/11216), measured by its authors on NISAR data). Formats move, and so do the libraries. + +Not yet possible does not mean never possible. Re-evaluate the options to pinpoint what needs to change to make cloud-optimized access possible. This could mean revaluating reprocessing, conversion and virtualization. + +## Optional: make it quantitative + +The two stages above get you to a short list. When two or three candidates remain and the choice is close, a transparent score can make the trade-off explicit and quantitative, reducing bias. + +[`cng-formats-benchmark`](https://github.com/developmentseed/cng-formats-benchmark) is an open-source (MIT) harness behind the numbers on the [Case Studies](case-studies.qmd) page. Per format and per dataset it measures the object-size distribution and tier fitness, write throughput, range-request-aware read latency, and tile-display latency. You describe your own datasets in config files, so no provider or dataset is baked into the code, and you run it on your own infrastructure against your own data. See the [documentation](https://developmentseed.github.io/cng-formats-benchmark/) and the [configuration guide](https://developmentseed.github.io/cng-formats-benchmark/configuration/). + +## Worked examples + +Six products run through the framework on the [Case Studies](case-studies.qmd) page. Five are backed by measurements. + +One lesson recurs often enough to state up front: **going cloud-optimized is rarely smaller than the source you started from.** Overviews, levels of detail, and spatial indexes are structures the source did not carry, so they add volume, and that added volume is the feature you are buying. + +Where two candidates do differ in size, look at the overviews before you look at the codec. In the measured cases, a codec change moves the total by 0.3 % to 2 %. Adding an overview pyramid to a store that had none adds 28 % to 34 %. The pyramid also costs a COG only bytes, because its overviews sit inside the one file, and it costs a Zarr store objects, because each level is a separate array. + +## Acknowledgements + +This framework was developed as part of a study for the [Centre National d'Études Spatiales (CNES)](https://cnes.fr/), carried out by [Development Seed](https://developmentseed.org/). The case studies use public Copernicus and NASA missions, and the measurements come from that study's benchmark work. + +The framework builds on earlier work in this guide supported by [NASA IMPACT](https://impact.earthdata.nasa.gov/) (now [NASA ODSI](https://www.nasa.gov/marshall/marshall-space-flight-missions/office-of-data-science-and-informatics-odsi/)). See [Our Supporters](../contributing.qmd#thank-you-to-our-supporters). diff --git a/glossary.qmd b/glossary.qmd index c4c2069d..b84f398e 100644 --- a/glossary.qmd +++ b/glossary.qmd @@ -9,6 +9,12 @@ format: This glossary aims to describe all the jargon of geospatial in the cloud! Are we missing something? [Create an issue](https://github.com/cloudnativegeo/cloud-optimized-geospatial-formats-guide/issues/new?title=Glossary+suggestion:&labels=glossary) to suggest an improvement. +#### Access pattern + +What subsets of the data get requested: a time series at a single point, a spatial window at a single date, or a full scan of everything. + +The access pattern drives the [chunk](#chunk) shape and the order of the dimensions, which is a separate question from [chunk size](#chunk-size). A cube chunked long in time answers a time series at one point cheaply and a full map expensively; chunked wide in latitude and longitude it does the opposite. Only rewriting the data lets you change it, so a virtualized dataset inherits whatever access pattern its source was written for. + #### Amazon S3 (S3) The [object storage](#object-storage-cloud-storage) service offered by Amazon. Part of [Amazon Web Services](#amazon-web-services-aws). @@ -312,6 +318,10 @@ An R library for using and managing [vector data](#vector-data), organized aroun A [vector data](#vector-data) file format. There are [many reasons](http://switchfromshapefile.org/) to no longer use Shapefile. +#### Shard + +The definition of shard here is specific to [Zarr](#zarr). A shard is a single object in object storage that contains multiple [chunks](#chunk) of data. See also: [Zarr sharding](https://zarr.readthedocs.io/en/latest/user-guide/performance/#sharding). + #### Space-filling curve An algorithm that translates two- or n- dimensional data into one-dimensional data. In practice, this is used as part of [spatial indexes](#spatial-index) to group [vector geometries](#vector-data) nearby within a file according to their two-dimensional location. @@ -328,6 +338,12 @@ For further reading: [A dive into spatial search algorithms](https://blog.mapbox A file format for image and raster data that supports [lossless compression](#lossless-compression). +#### Tiered storage + +A storage service that keeps data on different classes of hardware at different prices and speeds. A common arrangement is a "warm" tier on disk for data that is read often, and a "cold" tier on tape or archival storage for data that is read rarely. + +Some providers set a recommended minimum mean [object](#object-storage-cloud-storage) size for each tier. The storage service holds one index entry for each object, and very many small objects can fill that index before the storage capacity is full. Some providers enforce this limit. Other providers give it as guidance only. Find the policy of the service that you use, then make your [chunk size](#chunk-size) and grouping choices against it. + #### Vector data A type of geospatial data to represent points, lines, and polygons. diff --git a/index.qmd b/index.qmd index 01be62f5..ec88206e 100644 --- a/index.qmd +++ b/index.qmd @@ -64,7 +64,9 @@ Notes: ## Table of Contents 1. [Overview of Formats (slideshow)](./overview.qmd) -2. Formats +2. [Format Selection Decision Framework](./decision-framework/index.qmd) + a. [Case Studies](./decision-framework/case-studies.qmd) +3. Formats a. [Cloud-Optimized GeoTIFFs](./cloud-optimized-geotiffs/intro.qmd) b. [Zarr](./zarr/intro.qmd) c. [Cloud-Optimized NetCDF4/HDF5](./cloud-optimized-netcdf4-hdf5/index.qmd) @@ -72,7 +74,7 @@ Notes: e. [GeoParquet](./geoparquet/index.qmd) f. [FlatGeobuf](./flatgeobuf/intro.qmd) g. [PMTiles](./pmtiles/intro.qmd) -3. [Cookbooks](./cookbooks/index.qmd) +4. [Cookbooks](./cookbooks/index.qmd) ## Running Examples @@ -85,6 +87,7 @@ Most of the data formats covered in this guide have a Jupyter Notebook example t * [Alex Mandel](https://developmentseed.org/team/alex-mandel) * [Kyle Barron](https://github.com/kylebarron) * [Zac Deziel](https://developmentseed.org/team/zac-deziel) +* [Emmanuel Mathot](https://developmentseed.org/team/emmanuel-mathot) * [Overview Slide](./overview.qmd) credits: Vincent Sarago, Chris Holmes, Patrick Quinn, Matt Hanson, Ryan Abernathey ## Questions to Ask When Generating Cloud-Optimized Geospatial Data in Any Format