diff --git a/decision-framework/case-studies.qmd b/decision-framework/case-studies.qmd index 4f98617..20919da 100644 --- a/decision-framework/case-studies.qmd +++ b/decision-framework/case-studies.qmd @@ -141,7 +141,7 @@ The intent mix makes this case a genuine two-way contest, though not in the way - **Streaming is a visualization property, not a download one.** FlatGeobuf [streams](../flatgeobuf/intro.qmd#streaming-features-is-supported), so features render progressively as they arrive. Valuable, but visualization is absent from this mix. - **FlatGeobuf wins on whole-row spatial subsetting.** It is row-oriented with a Hilbert R-tree, so every attribute of the features in a bounding box is one contiguous read, where a [column-oriented](../geoparquet/index.qmd#column-oriented) store has to touch every column. The measurements below put this at about 10x faster. It is a real access pattern, but it is not one that this intent mix asks for. -At volume the scale question would matter too, because GeoParquet shards across [multiple files](../geoparquet/index.qmd#multi-file-support) while FlatGeobuf is a single file with a single index. These passes are tiny, so it does not bite here. +At volume the scale question would matter too, because GeoParquet shards across [multiple files](../geoparquet/index.qmd#multi-file-support) while FlatGeobuf is a single file with a single index. The SWOT L2_HR passes are tiny, so the scale question does not matter here. So GeoParquet wins on download *and* fusion. The product is fixed-schema tabular, with a lake id, a reach id, water-surface elevation, area and more, which is the native fit for its columns and row groups. A sparse Zarr store would add complexity with no gain. diff --git a/decision-framework/index.qmd b/decision-framework/index.qmd index e6529cc..8092aff 100644 --- a/decision-framework/index.qmd +++ b/decision-framework/index.qmd @@ -7,14 +7,14 @@ subtitle: "How to go from data you must publish to a short list of cloud-optimiz This framework helps you go from *"I have geospatial data to publish or migrate"* to a **short list of cloud-optimized format candidates**, and to the **layout choices** that make each candidate work. -It is a guide, not an oracle: +It is a guide to help make informed decisions, and does not prescribe a single correct format. - **What you control** comes first: whether the data already exist, whether you may reprocess them, and whether a mandate fixes an archival format. This can settle the answer before format even comes up. -- The **shape of your data** narrows the field. +- The **shape of your data** narrows the set of candidate formats. - The **intended use** and a few practical levers rank the remaining options. - **You make the final call.** -The framework is provider-neutral. Where a concrete number helps, such as an object-size policy or a benchmark result, the number is given as a worked example and flagged as an example, not as a rule. +The framework is data- and cloud-provider-neutral. Within data and cloud-provider contexts, a rule may help make a decision. For example, an object-size policy or a benchmark result may influence the choice. These cases are given as worked examples, not rules. This is a first, deliberately simple version. It aligns on the questions to ask and the order to ask them in. Follow the links to a format page when you need more detail. @@ -35,11 +35,11 @@ The framework uses a few terms in a specific way. Each links to the section that | [**Access pattern**](../glossary.qmd#access-pattern) | What subsets readers will request: a time series at one point, a spatial window at one date, a full scan. Intent implies an access pattern, and your layout either serves it or fights it. Defined in the guide's glossary; the layout consequences are in [the grouping lever](#grouping-lever). | | [**Archival-format mandate**](#control) | A requirement to maintain a specific archival format for preservation, such as HDF5, netCDF or GRIB, whatever cloud-optimized format you publish alongside it. | | [**Control**](#control) | Whether the choice is yours to make at all: do the data already exist, and may you reprocess them, technically and legally. This is Stage 1 of the framework, and it can end in "not yet possible". | -| [**Ecosystem maturity**](#maturity) | How settled a format is, on an L1 to L5 scale from experimental to de-facto standard. It answers "safe default, or bet?". | -| [**Grouping lever**](#grouping-lever) | The knob each format gives you for grouping bytes into objects: COG internal tiling, Zarr v3 sharding, COPC octree nodes, GeoParquet row groups. | -| [**Intent**](#intent) | What the data are for: archive and download, processing, visualization, distribution, or data fusion. Most products serve several at once, in proportion. | +| [**Ecosystem maturity**](#maturity) | How settled a format is, on an L1 to L5 scale from experimental to de-facto standard (see below). It answers "safe default, or bet?". | +| [**Grouping lever**](#grouping-lever) | The knobs each format gives you for grouping bytes into data chunks and objects in cloud storage: COG internal tiling, Zarr v3 chunking and sharding, COPC octree nodes, GeoParquet row groups. | +| [**Intent**](#intent) | What the data are for: archive and download, processing, visualization, distribution, or data fusion. Most products serve several intents at once. | | [**Interpretation encoding**](#interpretation-encoding) | Whether the format carries units, CRS, nodata and any scale and offset transform in-format, so every reader recovers the same physical value, or leans on a side catalog. | -| [**Object size**](#grouping-lever) | The size of the objects your layout actually writes to storage. Object stores reward a larger average object, and many apply a minimum size policy or sort objects into storage tiers. | +| [**Object size**](#grouping-lever) | The size of the objects your layout actually writes to storage. Object stores reward a larger average object and many apply a minimum size policy. | | [**Refinement**](#refinements) | A criterion applied *inside* a format family to rank its members. There are four: intent, the grouping lever, ecosystem maturity, and interpretation encoding. | | [**Topology**](#topology) | The shape of the data: 2D raster, N-D cubes, point cloud, or vector and tabular. Answering *what shape are the data?* selects a format family, not a format. | : {tbl-colwidths="[15,85]"} @@ -48,7 +48,7 @@ The framework uses a few terms in a specific way. Each links to the section that This guide already defines what makes a format cloud-optimized. The Welcome page states it as a four-part [cloud-optimized format pattern](../index.qmd#the-opportunity), and the glossary defines [cloud-optimized](../glossary.qmd#cloud-optimized) as the ability to read a meaningful *part* of a file without downloading *all* of it. -The framework does not restate that definition. It assumes it. Two consequences are worth naming here, because they decide whether a candidate belongs in the framework at all. They are entry conditions, not questions in the flowchart. +The framework starts from the definition on the Welcome page. Two requirements are worth explaining in more detail, because they decide whether a candidate belongs in the framework at all: **P1. The format must support partial and parallel reads over HTTP.** [Random access](../glossary.qmd#random-access) through [HTTP range requests](../glossary.qmd#http-range-request) is what makes a format usable from [object storage](../glossary.qmd#object-storage-cloud-storage). It depends on the rest of the pattern: [metadata](../glossary.qmd#metadata) in a consistent place that a reader can load in a single read, and then addressable [chunks](../glossary.qmd#chunk), internal tiling, or both. A reader that cannot locate the bytes it needs from the metadata alone cannot fetch only those bytes. @@ -56,7 +56,7 @@ The framework does not restate that definition. It assumes it. Two consequences ## The two stages -The framework runs in **two stages**. The topology question is the point where Stage 1 ends and Stage 2 begins. +The framework has **two stages**: **Stage 1 — control: do you have a choice?** Do the data already exist? If they exist, are you able (technically, legally) to reprocess them? This stage can end @@ -66,17 +66,7 @@ is no format decision to make yet. **Stage 2 — topology and refinement: which format?** You reach this stage only when Stage 1 leaves you free to choose. Here the shape of the data selects a **format family**: the small group of cloud-optimized formats built for that -shape. A *family*, not a format. Two or three candidates normally come through, -and the refinements below rank them. - -The change from Stage 1 to Stage 2 needs a clear statement. A reader can -interpret the diagram as if the shape of the data gives the answer: - -| Before the topology question | After it | -| --- | --- | -| Do I have a cloud-optimized target at all? | Which of these targets suits me best? | -| The outcome can be "not yet possible" | Every remaining candidate works; you are ranking | -| Decided by ownership and reprocessing rights | Decided by intent, layout, maturity, encoding | +shape. The refinements below rank the options within a format family. Four refinements then rank the candidates: **intent**, the **object-size and grouping lever**, **ecosystem maturity**, and **interpretation encoding**. The @@ -126,11 +116,13 @@ flowchart TD ``` -### The control fork {#control} +## The control fork {#control} **No existing data.** You are designing a new product, so you can choose freely. Go to the [topology question](#topology). -**If an archival-format mandate applies,** it adds a second job, and it does not remove your freedom to choose how readers get access. Applying cloud-optimized practice to the archive itself is not optional: whatever the mandated format, make it as range-request-friendly as it can be. What is genuinely optional is what you do beyond that. The stronger, recommended option is to also publish a full cloud-optimized copy for readers, since a native copy gives real range access rather than the archive's own chunking. A lighter option is to skip the second copy and instead wrap the optimized archive in a [virtual Zarr](../zarr/virtual-zarr.qmd) layer: cheaper to maintain, but you inherit the archive's chunk shape, the same limit as [existing data you cannot reprocess](#control) below. Where the mandated format is HDF5 or netCDF, this guide has [specific guidance](../cloud-optimized-netcdf4-hdf5/index.qmd) on consolidated metadata and chunk sizing. For other archival formats the principle holds even where the guide has no page: optimize what you must keep, then choose deliberately how far you go for readers. +**If an archival-format mandate applies,** it adds a second job, and it does not remove your freedom to choose how readers get access. Whatever the mandated format, make it as range-request-friendly as it can be. Where the mandated format is HDF5 or netCDF, this guide has [specific guidance](../cloud-optimized-netcdf4-hdf5/index.qmd) on consolidated metadata and chunk sizing. What is genuinely optional is what you do beyond the archival format. + +You can wrap the optimized archive in a [virtual Zarr](../zarr/virtual-zarr.qmd) layer: cheaper to maintain, but you inherit the archive's chunk shape, the same limit as [existing data you cannot reprocess](#control) below. The stronger, recommended option is to also publish a cloud-optimized copy for readers, since a cloud-optimized copy is an opportunity to provide access beyond the archive's native chunking. **Existing data you can reprocess.** You control the production chain, so you can also choose freely. Go to the [topology question](#topology). @@ -140,9 +132,9 @@ flowchart TD **Neither.** Some sources cannot be repackaged or virtualized today. Examples are HDF4, encrypted files, and a single large blob with no internal structure, whether or not it's compressed. See [Not yet possible](#not-yet-possible). -#### How to tell which branch you are on +### How to tell which branch you are on -Whether you can reprocess or virtualize usually depends on the data and data project context. Answers to these questions are not judgement calls. +Whether you can reprocess or virtualize usually depends on the data and data project context. **You can reprocess** if you have the processor, the inputs, and the budget to re-run the production chain. Then format and layout are both free choices: go to @@ -177,14 +169,14 @@ technically and legally free to change format and still be held by what its user tools expect. See [Existing users and their tools](#existing-users). -### Topology: what shape are the data? {#topology} +## Topology: what shape are the data? {#topology} -The shape of the data selects a **format family**: the candidate set you refine within. It does not select a format. [Intent](#intent) is what decides between the members. +The shape of the data selects a **format family**: the set of format candidates the refinement process narrows down. [Intent](#intent) is what decides between the members. | Data shape | Format family | Where each member wins | | --- | --- | --- | | 2D raster, per scene or per tile | [COG](../cloud-optimized-geotiffs/intro.qmd), [Zarr](../zarr/intro.qmd), [PMTiles](../pmtiles/intro.qmd) | COG for distribution and visualization; Zarr if the product is really a cube; PMTiles for pre-rendered tile delivery | -| N-D cubes, for example `time × y × x × band` | [GeoZarr](../zarr/intro.qmd), [COG](../cloud-optimized-geotiffs/intro.qmd), [Cloud-Optimized HDF/NetCDF](../cloud-optimized-netcdf4-hdf5/index.qmd) | GeoZarr for analysis across the cube; COG per time step if slices are read alone; cloud-optimized HDF/netCDF where an archival format is mandated or existing tooling must keep working | +| N-D cubes, for example `time × y × x × band` | [[Geo]Zarr](../zarr/intro.qmd), [COG](../cloud-optimized-geotiffs/intro.qmd), [Cloud-Optimized HDF/NetCDF](../cloud-optimized-netcdf4-hdf5/index.qmd) | GeoZarr for analysis across the cube; COG per time step if slices are read alone; cloud-optimized HDF/netCDF where an archival format is mandated or existing tooling must keep working | | Point cloud | [COPC](../copc/index.qmd), [GeoParquet](../geoparquet/index.qmd) | COPC for spatial subsetting and level of detail; GeoParquet if columnar analytics dominate | | Vector or tabular | [GeoParquet](../geoparquet/index.qmd), [FlatGeobuf](../flatgeobuf/intro.qmd), [PMTiles](../pmtiles/intro.qmd) | GeoParquet for joins and analytics; FlatGeobuf for streaming download; PMTiles for tiled web delivery | @@ -193,7 +185,7 @@ The question is about topology, not about the format the data arrives in. A netC ## Refinements {#refinements} -Apply these within the candidate set your topology opened. +These refinements help rank candidate formats within a family. ### Intent {#intent} @@ -203,44 +195,37 @@ What are the data for? Archive and download, processing, visualization, distribu - **On-the-fly and batched processing pull the layout in opposite directions.** On-the-fly access rewards small chunks, fine overviews, and cache locality: a reader fetches a little and wants it back fast. Batched processing rewards large shards or row groups, so that a bulk scan becomes a few large contiguous reads that many workers can run in parallel, instead of thousands of small scattered ones. - **Data fusion raises the stakes on encoding.** Combining products from different missions or processors means every input must be read back as the same physical value, and subsets from each must line up. Fusion is a form of processing, but it is worth naming separately: it weights [interpretation encoding](#interpretation-encoding) and consistent grids far higher than single-product analysis does, and it usually rewards a tabular or columnar target where joins are cheap. -Most real datasets serve several intents at once. Weigh them in proportion rather than picking one. The [case studies](case-studies.qmd) state an intent mix for each product, which is what decides the ranking inside a candidate set. - -### The object-size and grouping lever {#grouping-lever} +Most real datasets serve several intents at once. However, the intents often carry different weight in the decision process. The [case studies](case-studies.qmd) state an intent mix for each product, which is what decides the ranking inside a candidate set. -Two different controls are often both called the grouping lever. Keep them apart. One control sets the size of each object. The other control sets how finely a reader can address the bytes inside that object. +### The object size and grouping lever {#grouping-lever} -| Format | What sets the object size | What sets the read granularity inside the object | -| --- | --- | --- | -| COG | the extent of the file: area, bands, and time span | internal tiling and overviews | -| Zarr v3 | the shard size, because each shard is an object | the [chunk size](../glossary.qmd#chunk-size) inside the shard | -| COPC | the extent of the file: tile area and point density | octree nodes | -| GeoParquet | the partitioning: one file for each partition | row groups | +Every candidate format has a knob that sets how bytes are grouped into data chunks and chunks grouped into objects: -Only Zarr connects the two controls. In a Zarr store, each chunk or shard is itself an object. In the other formats, one file is one object. Internal tiling and row groups do not change the object size. - -Tune the lever to your [object store's](../glossary.qmd#object-storage-cloud-storage) **object-size policy** (S3, Azure Blob Storage, Google Cloud Storage, or an on-premises equivalent). Object stores degrade when they must index very many tiny objects, so most reward a larger average object. - -One worked example is the [CNES Datalake](https://cnes.fr/en/projects/centre-de-calcul), an in-house S3 service that moves data between disk and tape. Each [tier](../glossary.qmd#tiered-storage) recommends a minimum mean object size, measured as the total volume divided by the object count: about 32 MB for disk, and about 100 MB for tape. +| Format | Grouping lever | +| --- | --- | +| COG | internal tiling and overviews | +| Zarr v3 | chunks and shards ([chunk](../glossary.qmd#chunk) and [shard](../glossary.qmd#shard)) | +| COPC | octree nodes | +| GeoParquet | row groups, plus partitioning | -These floors are advisory, and nothing refuses your write. They protect the shared object index, which holds one entry for each object. Too many small objects fill that index before the disk is full, and the Datalake then cannot supply its pooled volume to any project. +Tune the lever to your [object store's](../glossary.qmd#object-storage-cloud-storage) **object-size policy** (S3, Azure Blob Storage, Google Cloud Storage, or an on-premises equivalent). Object stores degrade when they must index very many tiny objects, so most reward a larger average object. -Your own storage service has its own policy, or it has none. Find the policy, then set the lever against it. Some providers publish a threshold. Other providers apply a minimum billable object size instead. [Amazon S3](https://docs.aws.amazon.com/AmazonS3/latest/userguide/storage-class-intro.html) charges a minimum of 128 KB for each object in Standard-IA, One Zone-IA, and Glacier Instant Retrieval. A 10 KB object then costs the same as a 128 KB object. The write does not fail, so you do not see this cost until the invoice. +In general, when it comes to reading from object storage, objects can be too large (readers have to download and decompress more data than they need) or too small (readers make many small requests and pay the network latency for each). See also: [the datacube guide on tiny chunks](https://developmentseed.org/datacube-guide/latest/worst-practices/tiny-chunks.html) and [massive chunks](https://developmentseed.org/datacube-guide/latest/worst-practices/massive-chunks.html). -The correct direction depends on the size of your product: +One example: the [CNES Datalake](https://cnes.fr/en/projects/centre-de-calcul) sorts data into [tiers](../glossary.qmd#tiered-storage) by mean object size, with about 32 MB or more for warm disk and about 100 MB or more for cold tape. This serves the general point: find out whether your storage has an object-size policy, and adjust your groupings accordingly. -- **Products that are already large.** For Zarr, set the shard size so that the mean object size clears the policy. For a single-file format, the file is already one object. To make the objects smaller, split the file by region, by band, or by time step. -- **Products that are small.** Group *above* the product. Mosaic or concatenate many granules into one object that a reader can address with range requests. For Zarr, a larger shard does the same job. A finer internal layout inside one small granule makes the problem worse. +Some providers publish a threshold, some only charge you as if your objects were larger, and some have no policy at all. **Size and shape are two different questions.** How *big* the objects are is a question for your object store, answered above. How the bytes are arranged *inside* them (chunk shape, and the order of the dimensions) is an **access-pattern** question, -and the object store has no opinion on it. A `time × y × x` cube chunked long in time +and an object store provider has no opinion about the shape. A `time × y × x` cube chunked long in time serves a time series at one point cheaply and a full-scene map expensively; chunked wide in `y` and `x` it does the opposite. This guide works it through, with a diagram, in [chunk shape vs chunk size](../cloud-optimized-netcdf4-hdf5/index.qmd#chunk-shape-vs-chunk-size). Shape rarely decides *which* format you pick, because every candidate in a family lets -you set it: a COG chooses its tile dimensions, Zarr its chunk shape and dimension +you set the shape: a COG chooses its tile dimensions, Zarr its chunk shape and dimension order, GeoParquet its row-group size and row ordering. What shape decides is how well the format you picked performs, not which one you pick. And it is mostly settled back at the [control](#control) question, because only reprocessing lets you choose it freely. @@ -250,11 +235,11 @@ at the [control](#control) question, because only reprocessing lets you choose i Ecosystem maturity may influence the final decision. You may need something production-grade that you can count on, or you may have the flexibility to be an early adopter. Maturity is what separates a safe default from a bet. -To keep that judgement explicit rather than impressionistic, place each candidate on a five-level scale: +To keep that judgement explicit rather than impressionistic, this framework assigns candidate formats on the following five-level scale: | Level | Meaning | | --- | --- | -| **L5** | De-facto standard, universal tooling, no specification risk | +| **L5** | De-facto standard: universal tooling, no specification risk | | **L4** | Established: released spec, broad independent tooling, in production use | | **L3** | Stable: released spec, multiple implementations, growing tooling | | **L2** | Emerging: spec stabilizing, reference implementations, early adopters | @@ -266,6 +251,7 @@ Where the candidates sit today: | --- | --- | --- | --- | | [COG](../cloud-optimized-geotiffs/intro.qmd) | De-facto standard, on a GeoTIFF base | GDAL, rio-tiler, TiTiler, every GIS | **L5** | | [GeoParquet](../geoparquet/index.qmd) | v1.x, stable, on an Apache Parquet base | GDAL, GeoPandas, DuckDB, Arrow | **L4** | +| [Zarr](../zarr/intro.qmd) | v2 and v3 stable | GDAL, zarr-python, zarrs | **L4** | | [COPC](../copc/index.qmd) | v1.0, stable, on a LAZ base | PDAL, untwine, QGIS, copc.js | **L4** | | [FlatGeobuf](../flatgeobuf/intro.qmd) | Stable, on a flatbuffers base | GDAL and OGR, QGIS, flatgeobuf.js | **L3** | | [GeoZarr](../zarr/intro.qmd) | Draft, in an OGC Standards Working Group | Emerging, mostly xarray-based stacks | **L2** | @@ -278,31 +264,31 @@ Where the candidates sit today: This distinction is easy to miss, and it decides how much risk you actually take on. -- **Zarr v3, the substrate, is dependable.** The core specification was finalized in 2024. [Sharding](#grouping-lever), the feature that matters most for object storage, is stable in `zarr-python` 3.x and readable by GDAL and xarray. Judged on its own, the Zarr v3 core behaves like an L4. +- **Zarr v3, the substrate of GeoZarr, is dependable.** The core specification was finalized in 2024. [Sharding](#grouping-lever), the feature that matters most for object storage, is stable in `zarr-python` 3.x and readable by GDAL and xarray. Judged on its own, the Zarr v3 core behaves like an L4. - **GeoZarr, the convention layer, is still being ratified.** It is what makes a generic [Zarr](../glossary.qmd#zarr) store *geospatial*: CRS, dimension semantics, and multiscale pyramids. It does for Zarr roughly what the GeoTIFF tags do for a plain TIFF. It is a draft in an OGC Standards Working Group, on the standards track but not yet an adopted OGC standard, and the published draft can lag behind active convention work. Check [geozarr.org](https://geozarr.org/) for the current status before you rely on a specific version. -So the L2 above belongs to the convention layer, not to Zarr. The leading reference implementations today, such as ESA's EOPF Sentinel output, are demonstrators rather than production guarantees. +So the L2 above belongs to the convention layer, not to Zarr. The leading reference implementations of GeoZarr today, such as ESA's EOPF Sentinel output, are demonstrators rather than production guarantees. #### Maturity is a warning, not a veto -An L2 candidate can still be the right answer. The N-D cubes branch is exactly that case: there is no mature alternative with the same topology, so a low level does not mean "choose something else". It means **plan for change**. Budget for convention churn, pin the versions you depend on, and expect the visualization story in particular to improve as multiscale tooling catches up. +An L2 candidate can still be the right answer. Multidimensional cubes have no mature format candidate. A low level does not mean "choose something else". It means **plan for change**. Budget for convention churn, pin the versions you depend on, and expect visualization tooling in particular to improve as tooling catches up. -Read maturity against your own tolerance. A one-off internal product and a mission archive with a twenty-year retention commitment can reasonably land on different levels from the same candidate set. +Many factors influence what maturity level you can tolerate. A one-off internal product and a mission archive with a twenty-year retention commitment can reasonably land on different levels from the same set of candidate formats. ### Interpretation encoding {#interpretation-encoding} -Does the format carry the information a reader needs to recover the same physical value, or does it lean on a side catalog? The information is units, the [CRS](../glossary.qmd#coordinate-reference-system-crs), the nodata fill, and any scale and offset transform. +Does the format carry the information a reader needs to recover the same physical value, or does it lean on a side catalog? Interpretation encoding includes units, the [CRS](../glossary.qmd#coordinate-reference-system-crs), the nodata fill value (see also [Virtualizarr's distinction of fill values](https://virtualizarr.readthedocs.io/en/latest/explanation/custom_parsers.html#fill-values)), and any scale and offset transform. -This axis separates the candidates more than people expect: +Interpretation encoding separates the candidates more than people expect: -- **Zarr v3 makes the information first-class.** With the `scale_offset` codec, a declared fill value, Zarr conventions for projection, CRS and multiscale, the transform is a first-class array property. Every reader recovers the same value. -- **COG, GeoParquet, and COPC standardize the CRS**, but they do not standardize the value transform. A COG that holds scaled integers usually keeps its scale and offset in a GDAL metadata tag, or in a STAC property, or only in the product handbook. These locations are conventions. They are not part of the data model of the format. One reader applies the transform, and another reader returns the raw integers. +- **Zarr v3 makes the information first-class.** With the `scale_offset` codec, a declared fill value, Zarr conventions for projection, CRS and multiscales, the transform is a first-class array property. Every reader recovers the same value. +- **COG, GeoParquet, and COPC standardize the CRS** but leave the value transform to side metadata. Readers honor side metadata unevenly. -If your product stores scaled integers, or if an undeclared fill value would change a computed result, weigh this axis heavily. +If your product stores scaled integers, or if an undeclared fill value would change a computed result, weigh this aspect heavily. ## Cross-cutting considerations -These are not decision points. They apply whatever you choose. +These considerations do not impact what format you choose, but they do affect how you implement that format. ### Metadata and cataloging @@ -315,18 +301,20 @@ At a scale of tens of thousands to millions of files, collection-level discovery Some choices sit below the format and are foundational: - Base codecs, such as [ZSTD](../glossary.qmd#zstd) and [Deflate](../glossary.qmd#deflate) -- Data types and scale-offset encoding +- Data types, and scale-offset encoding - Sharding in Zarr v3 -These are mature and in scope. The per-format pages cover them. More experimental optimizations, such as bit-rounding and advanced compression tricks, are out of scope here, or should be marked as immature when you use them. +These are mature and in scope. The per-format pages cover them. More experimental optimizations, such as bit-rounding and advanced compression tricks, are out of scope at this time. Formats also differ in which data types they support, and in whether types can be mixed within one file, column, or schema. Those specifics live on each format's own page and are worth checking before committing to a candidate; they can rule one out as surely as topology or intent. ### Multiscale and native mixed resolution -A multiscale pyramid does not have to come from one source resolution downsampled by a fixed factor. The common case is **overviews**, also called [pyramids](../glossary.qmd#overviews): downsampled copies of the same band, generated at a uniform decimation so a reader can zoom out without reading and resampling full-resolution chunks. COG overviews, COPC's octree levels of detail, and most GeoZarr pyramids work this way, and it is already covered above under [ecosystem maturity](#maturity) and the [grouping lever](#grouping-lever). +Providing downsampled levels of original data in the source resolution is a common pattern for providing good performance, usually for visualization. A common example is COG overviews, which provide tiles at standard resolutions for mapping clients. A multiscale pyramid is a more general concept: a set of levels of detail, each with its own resolution, that can be used for visualization or analysis. -The same structure can also host bands that were never derived from each other and were simply captured at different native resolutions, as long as the steps between levels are chosen to land on each native resolution rather than a fixed factor. Sentinel-2 is the reference case, worked through in [case 1](case-studies.qmd) of the case studies: the 10 m, 20 m, and 60 m bands from the same sensor fold into one pyramid by aligning the decimation to each step (10 m to 20 m halves the resolution, 20 m to 60 m divides it by three), and the pyramid can continue past 60 m into ordinary overviews using the same mechanism. The [Zarr multiscales convention](https://github.com/zarr-conventions/multiscales) is built for exactly this: scale factors between levels are not required to be uniform, and its own reference examples use this Sentinel-2 layout. +Often these multiscale pyramids are generated at a uniform decimation so a reader can zoom out without reading and resampling full-resolution chunks. COG overviews, COPC's octree levels of detail, and most GeoZarr pyramids work this way, and it is already covered above under [ecosystem maturity](#maturity) and the [grouping lever](#grouping-lever). + +How these multiscale pyramids are generated can vary: Sometimes, data is provided at varying resolution where lower resolution data is not derived from the higher resolution data. Sentinel-2 is an example, worked through in [case 1](case-studies.qmd) of the case studies: Sentinel-2 collects data in 10m, 20m, and 60m bands from the same sensor. These bands can become a pyramid by aligning the decimation to each step (10m to 20m halves the resolution, 20m to 60m divides it by three), and the pyramid can continue past 60 m into ordinary overviews using the same mechanism. [Zarr multiscales convention](https://github.com/zarr-conventions/multiscales) allows for non-uniform scale factors. This still interacts with the [grouping lever](#grouping-lever), since each native resolution needs its own tiling or chunk shape, and with [interpretation encoding](#interpretation-encoding), since each band needs its own geotransform. What changes from the plain-overview case is that these become refinements within one multiscale structure. @@ -348,17 +336,15 @@ In the short term the answer is usually the same: publish the cloud-optimized fo When a source cannot be repackaged or virtualized today, the honest answer is *"not yet implemented"*, not *"impossible"*. The blocker is usually a missing parser, an unsupported codec, or source files with no internal chunking at all. Every one of those can change. -The precedent is HDF5 itself: once the standard example of a format that could not be read efficiently from object storage, until a change in write strategy and reader tooling made it viable, see [Cloud-Optimized HDF/NetCDF](../cloud-optimized-netcdf4-hdf5/index.qmd).The reader half is still moving: switching xarray's default cache for the h5netcdf engine improves dataset opening ([xarray #11216](https://github.com/pydata/xarray/pull/11216), measured by its authors on NISAR data). Formats move, and so do the libraries. +The precedent is HDF5 itself: once the standard example of a format that could not be read efficiently from object storage, until a change in write strategy and reader tooling made it viable, see [Cloud-Optimized HDF/NetCDF](../cloud-optimized-netcdf4-hdf5/index.qmd). The reader half is still moving: switching xarray's default cache for the h5netcdf engine improves dataset opening ([xarray #11216](https://github.com/pydata/xarray/pull/11216), measured by its authors on NISAR data). Formats move, and so do the libraries. -Frame the branch that way, and route the reader back to the options: reprocess if you can get the chain changed, virtualize if a reader exists, or convert on ingest. Say what is missing, so somebody can unlock it. +Not yet possible does not mean never possible. Re-evaluate the options to pinpoint what needs to change to make cloud-optimized access possible. This could mean revaluating reprocessing, conversion and virtualization. ## Optional: make it quantitative -The two stages above give you a short list. If two or three candidates remain, and the choice is close, measure them. This section is **optional**. The framework works without it. - -[`cng-formats-benchmark`](https://github.com/developmentseed/cng-formats-benchmark) is an open-source harness for this task. You describe your datasets in configuration files. The harness then measures the object-size distribution, the write throughput, the read latency, and the tile-display latency for each format. It also gives an optional weighted score, so you can rank the candidates against your own intent mix and read the margin to the second candidate. The numbers on the [Case Studies](case-studies.qmd) page come from this harness. +The two stages above get you to a short list. When two or three candidates remain and the choice is close, a transparent score can make the trade-off explicit and quantitative, reducing bias. -Two points are easy to get wrong if you score the candidates. Keep the object size and the tier fit beside the score, and not inside it, because a storage policy is a property of your infrastructure and not of the format. Set the weights yourself, because they belong to your context and not to the data. The [harness documentation](https://developmentseed.github.io/cng-formats-benchmark/) gives the details. +[`cng-formats-benchmark`](https://github.com/developmentseed/cng-formats-benchmark) is an open-source (MIT) harness behind the numbers on the [Case Studies](case-studies.qmd) page. Per format and per dataset it measures the object-size distribution and tier fitness, write throughput, range-request-aware read latency, and tile-display latency. You describe your own datasets in config files, so no provider or dataset is baked into the code, and you run it on your own infrastructure against your own data. See the [documentation](https://developmentseed.github.io/cng-formats-benchmark/) and the [configuration guide](https://developmentseed.github.io/cng-formats-benchmark/configuration/). ## Worked examples @@ -372,4 +358,4 @@ Where two candidates do differ in size, look at the overviews before you look at This framework was developed as part of a study for the [Centre National d'Études Spatiales (CNES)](https://cnes.fr/), carried out by [Development Seed](https://developmentseed.org/). The case studies use public Copernicus and NASA missions, and the measurements come from that study's benchmark work. -The framework builds on earlier work in this guide supported by [NASA IMPACT](https://impact.earthdata.nasa.gov/). See [Our Supporters](../contributing.qmd#thank-you-to-our-supporters). +The framework builds on earlier work in this guide supported by [NASA IMPACT](https://impact.earthdata.nasa.gov/) (now [NASA ODSI](https://www.nasa.gov/marshall/marshall-space-flight-missions/office-of-data-science-and-informatics-odsi/)). See [Our Supporters](../contributing.qmd#thank-you-to-our-supporters). diff --git a/glossary.qmd b/glossary.qmd index 3885996..b84f398 100644 --- a/glossary.qmd +++ b/glossary.qmd @@ -318,6 +318,10 @@ An R library for using and managing [vector data](#vector-data), organized aroun A [vector data](#vector-data) file format. There are [many reasons](http://switchfromshapefile.org/) to no longer use Shapefile. +#### Shard + +The definition of shard here is specific to [Zarr](#zarr). A shard is a single object in object storage that contains multiple [chunks](#chunk) of data. See also: [Zarr sharding](https://zarr.readthedocs.io/en/latest/user-guide/performance/#sharding). + #### Space-filling curve An algorithm that translates two- or n- dimensional data into one-dimensional data. In practice, this is used as part of [spatial indexes](#spatial-index) to group [vector geometries](#vector-data) nearby within a file according to their two-dimensional location.