Skip to content

perf: optimize compute runtime and memory - #13

Draft
ceholden wants to merge 11 commits into
mainfrom
perf/algorithms
Draft

ceholden wants to merge 11 commits into
mainfrom
perf/algorithms

Conversation

@ceholden

@ceholden ceholden commented Nov 4, 2026

Copy link
Copy Markdown
Collaborator

No description provided.

@ceholden
ceholden added this pull request to stack #14 November 4, 2026 22:16
Base automatically changed from fix/sentinel2-saturation to main November 5, 2026 21:11
ceholden and others added 11 commits November 5, 2026 16:11
Peak memory on l9_clear drops from 12.71 to 10.97 GB and wall time from 328
to 302 s; s2 drops from 5.89 to 5.75 GB and 79 to 72 s. Final masks are
identical on all three test scenes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
UNet classify no longer copies the predictor bands before normalizing, and
normalization no longer copies the cube or scales whole bands in float64 at
once. Byte-for-byte comparison with v5.0.1-2 on synthetic data passed on
perf/equivalence-tests; final masks and timing are to be evaluated on AWS Batch.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Unit tests confirm the model receives the same matrix as v5.0.1-2; final masks
and timing are to be evaluated on AWS Batch.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Spectral bands are read and converted in up to four threads, and the sun angle
cosine is computed once. Observation mask, band and saturation updates stay in
the main thread in the original order. Byte-for-byte comparison with v5.0.1-2 on synthetic data passed on
perf/equivalence-tests; final masks and timing are to be evaluated on AWS Batch.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Byte-for-byte comparison with v5.0.1-2 on synthetic data passed on
perf/equivalence-tests; final masks and timing are to be evaluated on AWS Batch.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Spectral bands are read and converted in up to four threads, and saturation is
read from the DN in the same step. Observation mask, band and saturation updates
stay in the main thread in the original order.

Before rebasing on the saturation fix, a byte-for-byte comparison with v5.0.1-2
on synthetic data passed on perf/equivalence-tests. After it, read_datacube output
matched the fix branch byte for byte on a synthetic scene with saturated pixels.
Final masks and timing are to be evaluated on AWS Batch.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With raw values up to 3000 and normalize=False, logits reach about 1800, where one
float32 step can move a probability by more than the 1e-5 tolerance. Batches of
different shapes can round differently on x86, so the test failed in CI on Linux
while passing on macOS.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Nothing in the repository calls it, and it had drifted from classify.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each test file now follows the module it covers, so coverage is easier to reason
about: test_satellite.py for Data and the shared band reader, with
test_satellite_landsat.py and test_satellite_sentinel.py for each sensor, then
test_utils.py, test_reconstruct.py, test_lightgbmlib.py and test_unetlib.py.

The shared band reader tests now run against the Satellite base class instead of
a Landsat scene. The same 178 tests are collected as before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With values from 0 to 2, zero is the smallest value, so removing the signbit guard
in reconstruct_erosion left the results unchanged. Adding negative values puts
signed zeros mid-range, where the order of equal values matters, so the case now
fails without the guard.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Removes tests that repeat another test or pin incidental details, and trims
parametrization that adds cases without adding behavior, from 178 cases to 106.
Test_read_bands_limits_results_waiting_behind_a_slow_band is replaced by a test
that counts submitted reads after each result, so it needs no sleep.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant