fix(climate): replace 30-day rolling baseline with WMO 30-year normals#2561
fix(climate): replace 30-day rolling baseline with WMO 30-year normals#2561fuleinist wants to merge 4 commits intokoala73:mainfrom
Conversation
- Create seed-climate-zone-normals.mjs to fetch 1991-2020 historical monthly means from Open-Meteo archive API per zone - Update seed-climate-anomalies.mjs to use WMO normals as baseline instead of climatologically meaningless 30-day rolling window - Add 7 new climate-specific zones: Arctic, Greenland, WestAntarctic, TibetanPlateau, CongoBasin, CoralTriangle, NorthAtlantic - Register climateZoneNormals cache key in cache-keys.ts - Add fallback to rolling baseline if normals not yet cached Fixes: koala73#2467
- seed-climate-zone-normals.mjs: Now fetches normals for ALL 22 zones (15 original geopolitical + 7 new climate zones) instead of just the 7 new climate zones. The 15 original zones were falling through to the broken rolling fallback. - seed-climate-anomalies.mjs: Fixed rolling fallback to fetch 30 days of data when WMO normals are not yet cached. Previously fetched only 7 days, causing baselineTemps slice to be empty and returning null for all zones. Now properly falls back to 30-day rolling baseline (last 7 days vs. prior 23 days) when normals seeder hasn't run. - cache-keys.ts: Removed climateZoneNormals from BOOTSTRAP_CACHE_KEYS. This is an internal seed-pipeline artifact (used by the anomaly seeder to read cached normals) and is not meant for the bootstrap endpoint. Only climate:anomalies:v1 (the final computed output) should be exposed to clients. Fixes greptile-apps P1 comments on PR koala73#2504.
|
Someone is attempting to deploy a commit to the Elie Team on Vercel. A member of the Team first needs to authorize it. |
Greptile SummaryThis PR replaces the climatologically flawed 30-day rolling baseline in
Additional P2 observations:
Confidence Score: 4/5
Important Files Changed
Sequence DiagramsequenceDiagram
participant Cron as Railway Cron (monthly)
participant NS as seed-climate-zone-normals.mjs
participant OM as Open-Meteo Archive API
participant Redis as Upstash Redis
Cron->>NS: trigger (1st of month)
loop For each of 22 zones (currently: 30 calls/zone)
NS->>OM: GET /v1/archive (1 year per request × 30)
OM-->>NS: daily temp + precip data
end
NS->>NS: aggregate to monthly means (12 normals/zone)
NS->>Redis: SET climate:zone-normals:v1 (TTL 30d)
participant AS as seed-climate-anomalies.mjs (every 3h)
participant OM2 as Open-Meteo Archive API
AS->>Redis: GET climate:zone-normals:v1
alt Normals cached
Redis-->>AS: { zones: [...22 zones × 12 months] }
AS->>OM2: GET /v1/archive (last 7 days per zone)
OM2-->>AS: daily data (may be < 7 due to archive lag ⚠️)
AS->>AS: anomaly = currentMean − monthNormal.tempMean
AS->>Redis: SET climate:anomalies:v1 (TTL 3h)
else Normals not yet cached (fallback)
Redis-->>AS: null
AS->>OM2: GET /v1/archive (last 30 days per zone)
OM2-->>AS: daily data
AS->>AS: anomaly = last7d mean − prev23d mean (old logic)
AS->>Redis: SET climate:anomalies:v1 (TTL 3h)
end
Reviews (1): Last reviewed commit: "fix(climate): address greptile-apps revi..." | Re-trigger Greptile |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| const daysToFetch = hasNormals ? 7 : 30; | ||
| const startDate = new Date(Date.now() - daysToFetch * 24 * 60 * 60 * 1000).toISOString().slice(0, 10); |
There was a problem hiding this comment.
When hasNormals = true, daysToFetch is set to 7, which means the request window is exactly today-minus-7 to today. The Open-Meteo archive API (archive-api.open-meteo.com) typically lags by 3-5 days — values for the last few days return as null and are filtered out by the null-check loop (lines 125–130).
On any given run this means temps will contain only 2–4 valid entries instead of 7, causing temps.length < 7 (line 132) to trigger and fetchZone to return null for every single zone. All 22 zones return null, the MIN_ZONES check fires, and the seeder throws — meaning the WMO-normals path always fails in production until this is fixed.
The existing fallback (no-normals, daysToFetch = 30) was immune to this because slicing temps.slice(-7) on 25+ valid days is safe even when the last few are unavailable.
| const daysToFetch = hasNormals ? 7 : 30; | |
| const startDate = new Date(Date.now() - daysToFetch * 24 * 60 * 60 * 1000).toISOString().slice(0, 10); | |
| const daysToFetch = hasNormals ? 14 : 30; |
Then keep the temps.slice(-7) call unchanged — it will naturally pick the 7 most-recent valid days even if the last few days are missing from the archive.
There was a problem hiding this comment.
Your comment has been addressed: The 7-day fetch window has been changed to 14 days when hasNormals=true (daysToFetch = hasNormals ? 14 : 30). This ensures at least 7 valid data points are available for the slice(-7) call even when the Open-Meteo archive API lags by 3-5 days.
| for (let year = startYear; year <= endYear; year++) { | ||
| const yearStart = `${year}-01-01`; | ||
| const yearEnd = `${year}-12-31`; | ||
| const url = `https://archive-api.open-meteo.com/v1/archive?latitude=${zone.lat}&longitude=${zone.lon}&start_date=${yearStart}&end_date=${yearEnd}&daily=temperature_2m_mean,precipitation_sum&timezone=UTC`; |
There was a problem hiding this comment.
The loop fetches one year at a time for each of 22 zones: 30 years × 22 zones = 660 sequential HTTP requests. With a 100 ms delay between requests plus Open-Meteo response latency (typically 1-3 s), the total wall-clock time is roughly 30–60 minutes per run. Railway's default cron job timeout is ~10 minutes.
The comment on line 68 even notes that Open-Meteo supports date ranges, but then contradicts itself by fetching year-by-year. Open-Meteo's archive endpoint fully supports a 30-year span in a single call (start_date=1991-01-01&end_date=2020-12-31).
Change the loop body to a single per-zone fetch:
// Replace the year-by-year loop with a single request covering the full 30-year range
const url = `https://archive-api.open-meteo.com/v1/archive?latitude=${zone.lat}&longitude=${zone.lon}&start_date=1991-01-01&end_date=2020-12-31&daily=temperature_2m_mean,precipitation_sum&timezone=UTC`;
This reduces 660 calls to 22 calls (one per zone), cutting runtime from ~45 minutes to ~2 minutes.
There was a problem hiding this comment.
Your comment has been addressed: The year-by-year loop has been replaced with a single per-zone API call covering the full 1991-2020 range. This reduces 660 sequential calls to 22 calls, cutting runtime from ~45 minutes to ~2 minutes and resolving the Railway cron timeout issue.
| // All 22 zones — must match ALL_ZONES in seed-climate-anomalies.mjs | ||
| const ALL_ZONES = [...ZONES, ...CLIMATE_ZONES]; |
There was a problem hiding this comment.
Both seed-climate-zone-normals.mjs and seed-climate-anomalies.mjs define ZONES, CLIMATE_ZONES, and ALL_ZONES independently (with a comment saying "must be kept in sync"). If a zone is renamed in one file but not the other, normals?.find((n) => n.zone === zone.name) in the anomaly seeder returns undefined for that zone — silently falling back to the rolling 30-day baseline with no error or warning.
Consider extracting the zone definitions into scripts/_climate-zones.mjs and importing from both seeders. This removes the synchronisation burden entirely.
There was a problem hiding this comment.
Your comment has been addressed: Zone definitions have been extracted into scripts/_climate-zones.mjs which exports ZONES, CLIMATE_ZONES, ALL_ZONES, and MIN_ZONES. Both seed-climate-zone-normals.mjs and seed-climate-anomalies.mjs now import from this shared file, eliminating the synchronisation burden.
| runSeed('climate', 'zone-normals', CANONICAL_KEY, fetchAllZoneNormals, { | ||
| validateFn: validate, | ||
| ttlSeconds: CACHE_TTL, | ||
| sourceVersion: 'open-meteo-archive-wmo-normals', | ||
| }).catch((err) => { | ||
| const _cause = err.cause ? ` (cause: ${err.cause.message || err.cause.code || err.cause})` : ''; | ||
| console.error('FATAL:', (err.message || err) + _cause); | ||
| process.exit(1); | ||
| }); |
There was a problem hiding this comment.
cache-keys.ts not updated as described in the PR
The PR description (Step 4) states that climateZoneNormals: 'climate:zone-normals:v1' will be added to server/_shared/cache-keys.ts, but cache-keys.ts was not modified in this PR. The key string 'climate:zone-normals:v1' is currently only defined as local constants in the two seed scripts.
While this doesn't break runtime behaviour (the seed scripts define their own constant), it means any server-side code that tries to read this key using BOOTSTRAP_CACHE_KEYS.climateZoneNormals would fail at the TypeScript level. Per the project conventions in AGENTS.md, cache key strings should be registered in cache-keys.ts as a single source of truth.
There was a problem hiding this comment.
Your comment has been addressed: climateZoneNormals: 'climate:zone-normals:v1' has been added to BOOTSTRAP_CACHE_KEYS in server/_shared/cache-keys.ts, aligning with the project convention that cache key strings should be registered there as a single source of truth.
fuleinist
force-pushed
the
fix/climate-wmo-normals
branch
from
8f996bf to
cc576f5
Compare
…closes P2 #3012404004)
1 participant
Summary
This PR replaces the climatologically meaningless 30-day rolling baseline with proper WMO 30-year climatological normals (1991-2020) for climate anomaly detection.
Problem
The current implementation in
seed-climate-anomalies.mjscompares the last 7 days against the previous 23 days of the same 30-day window. This is wrong because:Solution
Step 1: New
seed-climate-zone-normals.mjsclimate:zone-normals:v1(TTL 30 days)Step 2: Updated
seed-climate-anomalies.mjsclimate:zone-normals:v1from Redis as baselineclimate:anomalies:v1cache key — fix in placeStep 3: Added 7 new climate zones
New zones for climate-specific monitoring:
Step 4: Cache key registration
Added
climateZoneNormals: 'climate:zone-normals:v1'toserver/_shared/cache-keys.ts.Testing
Related Issue
Fixes #2467