Skip to content
Merged
Show file tree
Hide file tree
Changes from 51 commits
Commits
Show all changes
54 commits
Select commit Hold shift + click to select a range
73c87cd
Add format-selection decision framework page and case studies
emmanuelmathot Aug 3, 2026
7e37124
Link cng-formats-benchmark as the quantitative enabler
emmanuelmathot Aug 3, 2026
cd12599
Reframe the second size lesson around overviews and indexes
emmanuelmathot Aug 3, 2026
0d3e0d2
Expand ecosystem maturity with an explicit scale and the Zarr/GeoZarr…
emmanuelmathot Aug 4, 2026
69e2a2f
Update .gitignore to exclude Quarto notebook files and enhance clarit…
emmanuelmathot Aug 4, 2026
ef5b027
Enhance decision framework documentation by clarifying intent categor…
emmanuelmathot Aug 4, 2026
37a076d
Update decision-framework/index.qmd
emmanuelmathot Aug 5, 2026
ae6dbd5
Update decision-framework/index.qmd
emmanuelmathot Aug 5, 2026
6904983
Update decision-framework/index.qmd
emmanuelmathot Aug 5, 2026
4413be7
Refine decision framework by clarifying the two-stage process for for…
emmanuelmathot Aug 5, 2026
fc422dc
Enhance decision framework documentation by adding key terms and clar…
emmanuelmathot Aug 5, 2026
b9a385e
Clarify terminology and references in decision framework documentatio…
emmanuelmathot Aug 5, 2026
c0b8e41
Clarify decision framework by refining terminology and adding detaile…
emmanuelmathot Aug 5, 2026
a579973
Refine decision framework documentation by clarifying the interpretat…
emmanuelmathot Aug 5, 2026
f908d81
Enhance decision framework by expanding format family options for dat…
emmanuelmathot Aug 6, 2026
559c35f
Clarify decision framework by distinguishing between size and shape o…
emmanuelmathot Aug 6, 2026
a9ae111
Clarify terminology in decision framework by correcting pluralization…
emmanuelmathot Aug 6, 2026
2edda05
Clarify decision framework terminology by updating "unlock this" to "…
emmanuelmathot Aug 10, 2026
5a3443c
Clarify virtualization limitations in decision framework by updating …
emmanuelmathot Aug 10, 2026
dda3e36
Enhance decision framework by adding a repackage option for existing …
emmanuelmathot Aug 10, 2026
f5d684b
Clarify processing intent in decision framework by enhancing descript…
emmanuelmathot Aug 10, 2026
d9ac010
Update decision-framework/index.qmd
emmanuelmathot Aug 10, 2026
5bd680d
link
emmanuelmathot Aug 10, 2026
94bdb55
Update decision-framework/case-studies.qmd
emmanuelmathot Aug 10, 2026
b2d7ebf
Enhance decision framework documentation by adding access pattern def…
emmanuelmathot Aug 10, 2026
cba7555
Refine case studies in decision framework by adding detailed intent, …
emmanuelmathot Aug 10, 2026
f4e3bf3
Clarify the precedent of HDF5 in the decision framework by detailing …
emmanuelmathot Aug 10, 2026
45d16ca
Enhance case study on spatial indexing by clarifying the three-dimens…
emmanuelmathot Aug 10, 2026
e7fc113
Refine case study on GeoParquet and FlatGeobuf by clarifying download…
emmanuelmathot Aug 10, 2026
754932e
Update decision-framework/index.qmd
emmanuelmathot Aug 10, 2026
d8bc1ac
Update decision-framework/case-studies.qmd
emmanuelmathot Aug 10, 2026
35ecb16
Update decision-framework/case-studies.qmd
emmanuelmathot Aug 10, 2026
c58814f
Clarify product description and virtualization conditions for large i…
emmanuelmathot Aug 10, 2026
55aeb64
Add clarification on data control factors in decision framework
emmanuelmathot Aug 10, 2026
79187c0
Refine definition of "Access pattern" for clarity in glossary and dec…
emmanuelmathot Aug 10, 2026
01e395b
Update decision-framework/case-studies.qmd
emmanuelmathot Aug 10, 2026
eb6b7c3
Update decision-framework/case-studies.qmd
emmanuelmathot Aug 10, 2026
13820f5
Update decision-framework/index.qmd
emmanuelmathot Aug 10, 2026
f92d87e
Update decision-framework/case-studies.qmd
emmanuelmathot Aug 10, 2026
c665686
Update decision-framework/index.qmd
emmanuelmathot Aug 10, 2026
30bba09
Update decision-framework/index.qmd
emmanuelmathot Aug 10, 2026
ad1686f
Clarify example of object-size policy in storage backend section
emmanuelmathot Aug 10, 2026
b9b1327
Clarify the role of shape in format selection and its impact on perfo…
emmanuelmathot Aug 10, 2026
0008534
Clarify terminology by replacing "spine" with "framework" in decision…
emmanuelmathot Aug 10, 2026
5df3ade
Update decision-framework/index.qmd
emmanuelmathot Aug 10, 2026
afb32a0
Clarify terminology by replacing "backend" with "object store" in dec…
emmanuelmathot Aug 10, 2026
1323fa3
Update decision-framework/index.qmd
emmanuelmathot Aug 10, 2026
de216a8
Update decision-framework/case-studies.qmd
emmanuelmathot Aug 10, 2026
315dcc2
Clarify benchmark examples and terminology in case studies section
emmanuelmathot Aug 10, 2026
d142f07
Clarify GeoZarr store description and timing numbers in case studies …
emmanuelmathot Aug 10, 2026
8e04630
Remove benchmark caveat from GeoZarr case study for clarity
emmanuelmathot Aug 10, 2026
3405a29
Update decision-framework/index.qmd
emmanuelmathot Aug 21, 2026
ae41b39
Add clarification on Sentinel-2 multiscale dimensions and enhance ter…
emmanuelmathot Aug 27, 2026
2ad4bee
Clarify decision framework stages and object size policies in documen…
emmanuelmathot Sep 9, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -15,3 +15,5 @@ __pycache__/
.ipynb_checkpoints

/.luarc.json

**/*.quarto_ipynb
2 changes: 2 additions & 0 deletions CITATION.cff
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,8 @@ authors:
family-names: "Barron"
- given-names: "Zachary"
family-names: "Deziel"
- given-names: "Emmanuel"
family-names: "Mathot"
- given-names: "Max"
family-names: "Jones"
- given-names: "Brianna"
Expand Down
11 changes: 10 additions & 1 deletion _quarto.yml
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,15 @@ website:
text: Welcome
- href: overview.qmd
text: Overview Slides
# NOTE for @abarciauskas-bgse: proposed nav placement only.
# Placed before "Formats" so a reader can pick a format family before
# diving into a format page. Final position depends on the Track-3 site
# redesign - please confirm or move (and rename the path if needed).
- section: Choosing a Format
href: decision-framework/index.qmd
contents:
- decision-framework/index.qmd
- decision-framework/case-studies.qmd
- section: Formats
contents:
- section: Cloud Optimized GeoTIFFs (COG)
Expand Down Expand Up @@ -89,7 +98,7 @@ format:
html:
include-in-header: plausible.html
theme:
light: [cosmo, theme.scss] # from https://github.com/sta210-s22/website/blob/main/_quarto.yml
light: cosmo # from https://github.com/sta210-s22/website/blob/main/_quarto.yml
dark: [cosmo, theme-dark.scss]
code-copy: true
code-overflow: wrap
Expand Down
4 changes: 3 additions & 1 deletion _thankyous.qmd
Original file line number Diff line number Diff line change
Expand Up @@ -10,4 +10,6 @@ This guide has been made possible through the support of:
[![](./images/thankyous-devseed-logo.png){fig-alt="Development Seed Logo" fig-align="left" width=40%}](https://developmentseed.org)

[![](./images/thankyous-cng-logo.png){fig-alt="Cloud-Native Geospatial Forum" fig-align="left" width=40%}](https://cloudnativegeo.org)
:::
:::

The [Format Selection Decision Framework](./decision-framework/index.qmd) and its case studies were contributed from a study for the [Centre National d'Études Spatiales (CNES)](https://cnes.fr/), and use public [Copernicus](https://www.copernicus.eu/) and [NASA](https://www.earthdata.nasa.gov/) mission data.
184 changes: 184 additions & 0 deletions decision-framework/case-studies.qmd

Large diffs are not rendered by default.

370 changes: 370 additions & 0 deletions decision-framework/index.qmd

Large diffs are not rendered by default.

12 changes: 12 additions & 0 deletions glossary.qmd
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,12 @@ format:

This glossary aims to describe all the jargon of geospatial in the cloud! Are we missing something? [Create an issue](https://github.com/cloudnativegeo/cloud-optimized-geospatial-formats-guide/issues/new?title=Glossary+suggestion:&labels=glossary) to suggest an improvement.

#### Access pattern

What subsets of the data get requested: a time series at a single point, a spatial window at a single date, or a full scan of everything.

The access pattern drives the [chunk](#chunk) shape and the order of the dimensions, which is a separate question from [chunk size](#chunk-size). A cube chunked long in time answers a time series at one point cheaply and a full map expensively; chunked wide in latitude and longitude it does the opposite. Only rewriting the data lets you change it, so a virtualized dataset inherits whatever access pattern its source was written for.

#### Amazon S3 (S3)

The [object storage](#object-storage-cloud-storage) service offered by Amazon. Part of [Amazon Web Services](#amazon-web-services-aws).
Expand Down Expand Up @@ -328,6 +334,12 @@ For further reading: [A dive into spatial search algorithms](https://blog.mapbox

A file format for image and raster data that supports [lossless compression](#lossless-compression).

#### Tiered storage

A storage service that keeps data on different classes of hardware at different prices and speeds. A common arrangement is a "warm" tier on disk for data that is read often, and a "cold" tier on tape or archival storage for data that is read rarely.

Some providers set a minimum mean [object](#object-storage-cloud-storage) size for each tier, because indexing very many small objects is expensive. A layout whose objects fall below that floor can be refused or moved to a slower tier, so [chunk size](#chunk-size) and grouping choices have to be made against the policy of the storage you are actually using.

#### Vector data

A type of geospatial data to represent points, lines, and polygons.
Expand Down
7 changes: 5 additions & 2 deletions index.qmd
Original file line number Diff line number Diff line change
Expand Up @@ -64,15 +64,17 @@ Notes:
## Table of Contents

1. [Overview of Formats (slideshow)](./overview.qmd)
2. Formats
2. [Format Selection Decision Framework](./decision-framework/index.qmd)
a. [Case Studies](./decision-framework/case-studies.qmd)
3. Formats
a. [Cloud-Optimized GeoTIFFs](./cloud-optimized-geotiffs/intro.qmd)
b. [Zarr](./zarr/intro.qmd)
c. [Cloud-Optimized NetCDF4/HDF5](./cloud-optimized-netcdf4-hdf5/index.qmd)
d. [Cloud-Optimized Point Clouds (COPC)](./copc/index.qmd)
e. [GeoParquet](./geoparquet/index.qmd)
f. [FlatGeobuf](./flatgeobuf/intro.qmd)
g. [PMTiles](./pmtiles/intro.qmd)
3. [Cookbooks](./cookbooks/index.qmd)
4. [Cookbooks](./cookbooks/index.qmd)


## Running Examples
Expand All @@ -85,6 +87,7 @@ Most of the data formats covered in this guide have a Jupyter Notebook example t
* [Alex Mandel](https://developmentseed.org/team/alex-mandel)
* [Kyle Barron](https://github.com/kylebarron)
* [Zac Deziel](https://developmentseed.org/team/zac-deziel)
* [Emmanuel Mathot](https://developmentseed.org/team/emmanuel-mathot)
* [Overview Slide](./overview.qmd) credits: Vincent Sarago, Chris Holmes, Patrick Quinn, Matt Hanson, Ryan Abernathey

## Questions to Ask When Generating Cloud-Optimized Geospatial Data in Any Format
Expand Down
Loading