Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
57 commits
Select commit Hold shift + click to select a range
f6241af
Add zone-wide Zarr format for consolidated UTM zone stores
avsm Feb 22, 2026
de8cfbb
zarr-build: add Rich progress bars for scanning and writing
avsm Feb 22, 2026
dc5c7b8
zarr-build: parallelise tile scanning with multiprocessing
avsm Feb 22, 2026
67e57f5
zarr-build: compute expected tile grid from lon/lat and warn on mismatch
avsm Feb 22, 2026
fdcd40d
Fix zarr v3 codec API: use compressors= with ZstdCodec
avsm Feb 22, 2026
58c8a06
Fix compute_tile_grid to match landmask TIFF generation
avsm Feb 22, 2026
15e95b7
zarr-build: add dimension_names for xarray coordinate auto-detection
avsm Feb 22, 2026
4da6a47
zarr-build: warn when tiles straddle UTM zone boundaries
avsm Feb 22, 2026
704e7cb
Add browser-based Zarr embedding viewer with `geotessera-registry serve`
avsm Feb 23, 2026
c829ab8
serve: bind to 0.0.0.0 for network access
avsm Feb 23, 2026
70a438a
serve: auto-discover stores and add store selector dropdown
avsm Feb 23, 2026
1a28cd8
serve: use relative URLs to avoid CORS errors on remote hosts
avsm Feb 23, 2026
044770b
serve: generate chunk manifests to avoid 404s on sparse stores
avsm Feb 23, 2026
e09b26b
viewer: add chunk grid and UTM zone overlays, prioritise viewport-cen…
avsm Feb 23, 2026
853ccab
viewer: fix raster layer ordering and add console logging
avsm Feb 23, 2026
2a290c1
viewer: fix rendering, cancel stale loads, cap chunks per update
avsm Feb 23, 2026
7d39979
viewer: fix layer ordering with moveLayer, auto-zoom to first chunk
avsm Feb 23, 2026
f2e2a8e
zarr: add RGB preview array for fast false-colour overview
avsm Feb 23, 2026
0a4a605
zarr: fix chunk shape access for zarr v3 API
avsm Feb 23, 2026
c762752
zarr: parallel RGB stretch computation with threaded chunk reads
avsm Feb 23, 2026
e026382
zarr: fix NaN cast warning and parallelise RGB write pass
avsm Feb 23, 2026
6b990d2
zarr: parallelise tile writing and add --workers / -j flag
avsm Feb 23, 2026
4f906ec
zarr: respect --zones filter in --rgb-only mode
avsm Feb 23, 2026
41c1360
viewer: add WebWorker pool, AbortController, and progress bar
avsm Feb 23, 2026
cdfd5c0
zarr: add server-side PCA preview (pca_rgb array)
avsm Feb 23, 2026
643a365
faster pca sampling
avsm Feb 23, 2026
0ee6b79
zarr: faster tile scanning with progress and zone pre-filtering
avsm Feb 23, 2026
638d058
viewer: switch to zarrita.js for native sharding support
avsm Feb 23, 2026
3ea1fb4
zarr: shard-batched writes to avoid read-modify-write amplification
avsm Feb 23, 2026
643b194
zarr: remove sharding and compression, keep zarrita.js viewer
avsm Feb 24, 2026
4f995d3
zarr: use zarrs Rust codec pipeline when available
avsm Feb 24, 2026
f6caff2
zarr: use zarrs Rust pipeline for reads only, Python for writes
avsm Feb 24, 2026
2a6d741
zarr: make RGB and PCA previews opt-in (--rgb, --pca)
avsm Feb 24, 2026
a615d21
zarr: chunk-batched writes to eliminate read-modify-write
avsm Feb 24, 2026
8523b49
zarr: remove zarrs Rust codec pipeline
avsm Feb 24, 2026
40314f6
zarr: fix zarrita.js CDN version and add --store flag to serve
avsm Feb 24, 2026
3077d00
zarr: use absolute URLs for zarrita FetchStore
avsm Feb 24, 2026
6cb0439
zarr: simplify write pipeline, deduplicate code, improve PCA colours
avsm Feb 24, 2026
a823b57
zarr: auto-detect registry-dir from base_dir
avsm Feb 24, 2026
e72bc0b
zarr: merge file-existence check into tile scanning phase
avsm Feb 24, 2026
560cb1a
zarr: replace tile scanning with computed grid info
avsm Feb 24, 2026
20e78f1
zarr: eliminate file I/O from gather_tile_infos
avsm Feb 24, 2026
a7a4300
zarr: fix PCA SVD crash from non-finite scale values
avsm Feb 25, 2026
9e4b43f
zarr: sanitize inf scales to NaN throughout pipeline
avsm Feb 25, 2026
5e2b71d
docs: add STAC index command design
avsm Feb 25, 2026
e95d331
registry: add stac-index command for static STAC catalog generation
avsm Feb 25, 2026
7106aeb
stac-index: use UTM zone boundaries for longitude extent
avsm Feb 25, 2026
3a3adaa
docs: add UTM-native pyramid design for RGB/PCA previews
avsm Feb 25, 2026
6e58569
docs: add pyramid implementation plan
avsm Feb 25, 2026
746f1a3
zarr: add build_preview_pyramid for multi-resolution overviews
avsm Feb 25, 2026
13a010e
zarr: add add_pyramids_to_existing_store wrapper for pyramid CLI
avsm Feb 25, 2026
2fc0564
cli: add --pyramid and --pyramid-only flags to zarr-build
avsm Feb 25, 2026
188072e
docs: update zarr_zone module docstring with pyramid layout
avsm Feb 25, 2026
1435d24
tests: add pyramid building integration test
avsm Feb 25, 2026
14b66cc
zarr: improve pyramid progress bar and add parallel chunk writes
avsm Feb 25, 2026
85babb9
zarr: remove nullcontext, use explicit start/stop for progress
avsm Feb 25, 2026
6d30d30
zarr: fix pyramid OOM by using strip-based coarsening
avsm Feb 25, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
99 changes: 99 additions & 0 deletions docs/plans/2026-02-25-pyramid-design.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,99 @@
# UTM-native Pyramids for RGB/PCA Previews

## Problem

Current Zarr stores have single-resolution RGB and PCA preview arrays at 10m/px.
Browser clients must fetch full-resolution chunks even when zoomed out, wasting
bandwidth and making overview rendering slow for large UTM zones (~60k × 600k px).

## Design

Build multi-resolution pyramids of the existing `rgb` and `pca_rgb` RGBA arrays
using iterative 2× mean coarsening. Pyramids stay in UTM projection (no
reprojection to Web Mercator). The client selects the appropriate level based on
zoom and reprojects per-chunk as it does today.

### Store Layout

```
utm30_2025.zarr/
├── embeddings/ # unchanged
├── scales/ # unchanged
├── rgb/ # full-res RGBA uint8 (level 0)
├── pca_rgb/ # full-res RGBA uint8 (level 0)
├── rgb_pyramid/
│ ├── 1/ # 2× coarsened (20m)
│ ├── 2/ # 4× coarsened (40m)
│ ├── 3/ # 8× coarsened (80m)
│ ├── 4/ # 16× coarsened (160m)
│ ├── 5/ # 32× coarsened (320m)
│ ├── 6/ # 64× coarsened (640m)
│ └── 7/ # 128× coarsened (1280m)
├── pca_rgb_pyramid/
│ ├── 1/ … 7/
├── easting/
├── northing/
└── band/
```

Level 0 is the existing `rgb`/`pca_rgb` array — no duplication. Levels 1–7 are
stored under `{name}_pyramid/{level}/` as RGBA uint8 arrays with the same
1024×1024×4 chunking.

### Coarsening Method

Each level halves both spatial dimensions by averaging 2×2 blocks using
`xarray.DataArray.coarsen(northing=2, easting=2).mean()`. Alpha channel is
averaged too, correctly handling partial-data boundaries where some pixels in a
block are transparent.

8 fixed levels total (level 0 = full-res, levels 1–7 stored as pyramid groups).

### Per-Level Metadata

Each pyramid group stores attrs:

```python
{
"level": int, # pyramid level (1–7)
"pixel_size_m": float, # 10 * 2^level
"transform": [6 floats], # adjusted affine for this resolution
"shape": [height, width], # dimensions at this level
}
```

### Root Store Metadata

Added to existing store attrs:

```python
{
"has_rgb_pyramid": True,
"rgb_pyramid_levels": 8, # total including level 0
"has_pca_pyramid": True,
"pca_pyramid_levels": 8,
}
```

### CLI Integration

Two new flags on `geotessera-registry zarr-build`:

- `--pyramid` — build pyramids after writing previews (requires rgb/pca to exist)
- `--pyramid-only` — scan existing stores, build/rebuild pyramids from full-res
preview arrays without regenerating the full-res data

### Dependencies

No new dependencies. Uses `xarray.coarsen` (xarray is already a dependency).
ndpyramid's value is mainly for reprojection which we don't need since pyramids
stay in UTM.

### Implementation Notes

- Read full-res array as xarray DataArray, iteratively coarsen and write each level
- Chunk size stays 1024×1024×4 at all levels (coarser levels have fewer chunks)
- Uncompressed storage (consistent with existing preview arrays)
- Progress reporting via Rich (one progress bar per store per preview type)
- `--pyramid-only` scans for existing `.zarr` stores and checks `has_rgb_preview`
/ `has_pca_preview` attrs to decide what to build
Loading
Loading