Skip to content

eckit::geo: scanning orders, projections, search, caches and gridspec fixes - #374

Merged
pmaciel merged 82 commits into
developfrom
feature/eckit-geo
Oct 14, 2026
Merged

pmaciel merged 82 commits into
developfrom
feature/eckit-geo

Conversation

@pmaciel

@pmaciel pmaciel commented Oct 8, 2026 •

Copy link
Copy Markdown
Member

Description

These are eckit::geo fixes and features found while using gridspec grids for MIR regridding and for eccodes (regional, rotated and projected grids, scanning orders, GRIB grids), and the k-d tree search, caches and Python bindings built on them. The branch is organised as one merge per topic, in the order below; smaller changes are commits between the merges.

Longitude normalisation (fix/longitude-normalisation)

  • Angle normalisation: normalise_angle_to_minimum/maximum could return minimum + 360 for values just below the minimum (e.g. -8.9e-16 → 360). Results are now always in [min, min + 360). Same fix for PointLonLatR.
  • points_equal now treats points across the longitude seam as equal (e.g. 0 and 360 - 1e-12).
  • Regional regular_gg round trip: Gaussian latitude cropping now tolerates rounding, so a regional grid's spec rebuilds the same grid.
  • A longitude range cropped to a single value no longer asserts.

Regular grids and scanning orders (fix/regular-grids-scanning)

  • regular_ll: decreasing (i-/j-) periodic longitude ranges now start from the lowest point, like increasing ones.
  • regular_gg: i- orders were built on the wrong global range. Construction now uses helpers, which also fixes a leak of the cropped ranges.
  • order::Scan::reorder renumbers points between any two scanning orders (i±j±, j±i±).
  • Regular ranges end exactly at their limits (e.g. not past a pole).

Projections (fix/projections)

  • Rotation: south_pole_lon/south_pole_lat are individually optional. The Python test now expects this, which is the current behaviour; it used to expect an error.
  • test_projdb_fallback now runs only when the build actually depends on the proj.db bundled in the wheel. It skips when PROJ finds its own proj.db, e.g. a local build against a system PROJ.
  • PROJ thread safety: PROJ objects share a context, which isn't thread-safe. Creating, using and destroying them is now serialised, with a multi-threaded test.
  • Albers equal-area: the cone constant, and longitudes either side of the central meridian.

Search (feature/search)

  • cache::MemoryUsage (memory and shared memory footprints) and cache::InMemoryCache (capacity, footprint, evictions), as in MIR.
  • k-d tree search: nearest neighbour, k nearest neighbours, radius, and their union/intersection (knn_or_radius, knn_and_radius).
  • Grid::search(), a search shared through cache::SearchCache.
  • Python: the eckit.geo search module, grid search loaders and examples. eckit-geo-grid-nearest uses the grid search.

SPDX headers

  • The Cython sources (.pyx/.pxd) now carry SPDX headers, like all other sources. cython-lint allows 90 characters (the length of the SPDX copyright line), so the corresponding REUSE.toml exception is removed.

Python extension modules

  • eckit.geo is built as one extension module per class (area, figure, projection, grid, search) instead of a single one. The Python API is unchanged.

Cache layout (feature/cache-layout)

  • Cache versions are in one place (cache/Version.h): download, latlon and search.
  • All downloads (ORCA, ICON and FESOM coordinates, shapefiles) go to download/<version>/ under the eckit::geo cache directory. Coordinates written locally (e.g. unstructured grids) go to latlon/<version>/.
  • Downloads, unzipping and the latlon cache are safe across processes (through eckit::CacheManager): files are written to a temporary file and then renamed, so concurrent processes never read a partial file.
  • eckit-geo-grid-cache takes grid regular expressions or uids as arguments.
  • Python: cache_dir, and the Python tests cache to a temporary directory.

Grid specs (feature/gridspec)

  • Grid type from the gridspec (ORCA, ICON, regular_xy), canonical forms, routings and spec round trips (tests).
  • Area, figure and projection specs as part of a grid spec (Area::fill_grid_spec, Figure::fill_projection_spec, Projection::fill_grid_spec). Defaults are left out: the default area, the default projection and the default figure.
  • The area and projection name registries are separate from the grid one, so grid names (e.g. SMUFF-OPERA-2km) aren't taken for projection names.
  • Reverse projections: Projection::reverse(), and the spec says "reverse": true instead of renaming the type (reverse-<type>). The Python Projection has a reverse property.
  • Shapefile spec: the .dbf path was written under the shp key, and the name field is now name_field.
  • eckit::spec::Custom::make_from_value handles booleans.

Search caching and geocentric points (feature/search-caching)

  • Grid::to_xyz: geocentric coordinates of the grid points. Grids of (lon, lat) points are converted on the figure of their coordinates (Projection::source_figure, which for PROJ can differ from the target CRS figure), and geocentric grid points are returned as they are. Also in Python, as Grid.to_xyz().
  • Grid::to_latlons accepts PointLonLatR points, and raises a GridError for points that aren't (lon, lat).
  • Search builds its trees from Grid::to_xyz and converts (lon, lat) queries with LonLatToXYZ on the same figure, without PROJ (thread-safe queries, and no PROJ database needed). LonLatToXYZ is simplified, with a vectorised conversion.
  • Bounding boxes of projections whose source points aren't (lon, lat) (e.g. geocentric) are global.
  • Search loaders as in MIR: file-io, mmap, shmem and tmp-shmem, replacing the previous tree classes.
  • The search cache makes room before loading a tree (from size estimates). Its capacity is a CMake option, eckit_GEO_CACHE_SEARCH_CAPACITY ("in-process[,shared]" bytes, default 1 GiB in-process), and the resource eckit-geo-cache-search-capacity.
  • Threading on eckit threads, or on STL threads (util/mutex.h).

Comments and dead code

  • Commented-out code and the unused Projection::falseXY setter are removed.

Projections as PROJ (feature/projections-as-proj)

  • regular_xy grids from x and y ranges (any step, the stop included to a fraction of the step), false easting/northing, and projections from (x, y). The spec states its order unless it's the default.
  • Lambert conformal conic (lcc), Mercator (merc), polar stereographic (stere) and Lambert azimuthal equal-area (laea) as PROJ, on spheres and ellipsoids, with catalogue grids and grids as the GRIB examples (e.g. lambert_lam.grib2).
  • space_view (geostationary satellite view) as GRIB, compared to PROJ geos.
  • PROJ strings of projections (Projection::proj_str, rotation as ob_tran), and grid points on the projection figure.
  • Reverse projections of any type, owning the projection, and from spec ("reverse": true).
  • swisslv95 and swisslv03 as catalogue grids and named projections; a projection name as grid type.
  • Test tolerances from the point types (EPS) and the bounding box precision; projection tests to 1e-6, independent of the platform.

Grid cropping and shapefile areas (feature/grid-crop)

  • Bounding boxes periodic from any west, containing points at the edges.
  • Grids cropped by areas (bounding boxes, polygons), cached (cache::CropCache, capacity eckit_GEO_CACHE_CROP_CAPACITY, default 0).
  • Shapefile areas (by name or index) from files or URLs, and shapefile libraries of shared areas (GISCO, by url_prefix). Areas are compared by uid.

Cleanup

  • The unused option GEO_PROJECTION_PROJ_DEFAULT is removed (CMake, LibEcKitGeo, Rust eckit-sys).
  • Tests compare with EXPECT/EXPECT_NOT.

Scanning orders (feature/scanning-orders)

  • The default scanning order is the empty order (i+j-), so default orders are left out of specs.
  • Regular grids of alternating rows (or columns), as GRIB alternativeRowScanning: iterator::Regular and reorder support them.
  • Python: Grid.reorder, Grid.to_points (coordinate names to values) and point_coordinates.

GRIB grids (feature/grib-grids)

  • Regular Gaussian grids of any number of longitudes (ni, unless 4N).
  • Ranges of a single value are kept as given and without increment, so grids of a single point, row or column are supported.

Figures and Albers on ellipsoids (feature/albers-ellipsoid)

  • Figure::conformal() and Figure::authalic(): functions of the latitude (conformal and authalic latitudes, isometric and authalic q), moved from the projections into the figure, and used by lcc, merc, stere, laea and aea.
  • Surface areas of all figures from the authalic q: Figure::area() and Figure::area(BoundingBox) (spheres and ellipsoids alike). BoundingBox::area uses it.
  • Albers equal-area on spheres and ellipsoids, as PROJ aea.

Crop preserving scanning modes (feature/crop-scanning-modes)

  • Regular grids (regular_ll, regular_gg) are cropped as their own type, preserving the scanning mode (including alternating rows). Grid::crop returns the indices of the points in the area, in the order of the cropped grid points.

Smaller changes

  • The proj projection type is registered on demand, and only if the PROJ database is available, so ProjectionFactory::has_type("proj") alone tells whether it can be used.
  • GreatCircle::course computes its sines and cosines locally (util/sincos.h removed).

Changes for existing users

  • Cache directories: downloads and local coordinates move to download/1/ and latlon/1/. Existing caches under latlon/, fesom/ and shapefile/ aren't reused, so files are downloaded or written once again.
  • Reverse projection specs: "reverse": true instead of the reverse-<type> type name.
  • Specs leave out the default scanning order (i+j-).
  • Grids of a single point, row or column have specs without increment in that direction (e.g. "grid":[0,0]).
  • The CMake option GEO_PROJECTION_PROJ_DEFAULT is removed.
  • The proj projection type isn't registered without the PROJ database.
  • Search: the tree classes (search/Tree*) are replaced by the loaders (search/KDTree, FileIO, MappedFile, SharedMemory, …), and util/sincos.h is removed.

Tests

  • C++ and Python tests for each fix and feature, added with them.

Contributor Declaration

By opening this pull request, I affirm the following:

  • All authors agree to the Contributor License Agreement.
  • The code follows the project's coding standards.
  • I have performed self-review and added comments where needed.
  • I have added or updated tests to verify that my changes are effective and functional.
  • I have run all existing tests and confirmed they pass.

🌦️ >> Documentation << 🌦️
https://sites.ecmwf.int/docs/dev-section/eckit/pull-requests/PR-374

🤖 Generated with Claude Code

@pmaciel pmaciel changed the title Feature/eckit geo eckit::geo: regular grid scanning orders, reorder, and longitude seam comparison fixes Oct 8, 2026
@pmaciel pmaciel changed the title eckit::geo: regular grid scanning orders, reorder, and longitude seam comparison fixes eckit::geo: scanning orders, projections, search, caches and gridspec fixes Oct 10, 2026
@pmaciel pmaciel added the run-downstream-ci:2 Run downstream CI up to level 2 label Oct 14, 2026
@pmaciel
pmaciel marked this pull request as ready for review October 14, 2026 00:31
… trees in memory, mapped, cache file, shared memory, Grid::search and search cache
…efix), areas from libraries, areas compared by uid
…ion::Conformal) and authalic; the surface areas of all figures from the authalic
…ing mode (the indices as the cropped grid points, eg. of alternating rows)
…atabase is available, tests by ProjectionFactory::has_type
@codecov-commenter

codecov-commenter commented Oct 14, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 89.34535% with 402 lines in your changes missing coverage. Please review.
✅ Project coverage is 70.58%. Comparing base (87ba1e8) to head (8f672ee).

Files with missing lines Patch % Lines
src/eckit/geo/projection/AlbersEqualArea.cc 0.00% 36 Missing ⚠️
tests/geo/search_kdtree.cc 93.43% 27 Missing ⚠️
src/eckit/geo/Projection.cc 75.24% 25 Missing ⚠️
tests/geo/grid_regular_ll.cc 79.24% 22 Missing ⚠️
src/eckit/geo/Grid.cc 63.79% 21 Missing ⚠️
src/eckit/geo/search/MappedFile.cc 85.82% 19 Missing ⚠️
tests/geo/grid.cc 64.15% 19 Missing ⚠️
src/eckit/geo/cache/Crop.cc 11.11% 16 Missing ⚠️
src/eckit/geo/cache/CropCache.cc 72.41% 16 Missing ⚠️
tests/geo/download.cc 77.27% 15 Missing ⚠️
... and 38 more
Additional details and impacted files
@@             Coverage Diff             @@
##           develop     #374      +/-   ##
===========================================
+ Coverage    68.81%   70.58%   +1.77%     
===========================================
  Files         1190     1207      +17     
  Lines        63233    66076    +2843     
  Branches      4758     5065     +307     
===========================================
+ Hits         43511    46639    +3128     
+ Misses       19722    19437     -285     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@pmaciel
pmaciel merged commit a888463 into develop Oct 14, 2026
185 of 202 checks passed
@pmaciel
pmaciel deleted the feature/eckit-geo branch October 14, 2026 01:14
@pmaciel pmaciel mentioned this pull request Oct 30, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

run-downstream-ci:2 Run downstream CI up to level 2

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants