Skip to content

fix(climate): replace 30-day rolling baseline with WMO 30-year normals #2504

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
fuleinist wants to merge 4 commits into from
Closed

fix(climate): replace 30-day rolling baseline with WMO 30-year normals #2504

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
17cc354
fix(i18n): use data-font attribute instead of inline style for font f…
fuleinist Mar 26, 2026
90de00a
fix: remove stray double blank line in font-settings.ts
Mar 27, 2026
dcc547b
fix(climate): replace 30-day rolling baseline with WMO 30-year normals
Mar 29, 2026
164931e
fix(climate): address greptile-apps review comments on WMO normals PR
Mar 29, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
144 changes: 128 additions & 16 deletions scripts/seed-climate-anomalies.mjs
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,12 +1,25 @@
#!/usr/bin/env node

import { loadEnvFile, CHROME_UA, runSeed } from './_seed-utils.mjs';
/**
* seed-climate-anomalies.mjs
*
* Computes climate anomalies by comparing current 7-day means against
* WMO 30-year climatological normals (1991-2020) for the current calendar month.
*
* The previous approach of comparing against the previous 23 days of the same
* 30-day window was climatologically wrong — a sustained heat wave during a
* uniformly hot month would not appear anomalous because the baseline was
* equally hot.
*/

import { loadEnvFile, CHROME_UA, runSeed, getRedisCredentials } from './_seed-utils.mjs';

loadEnvFile(import.meta.url);

const CANONICAL_KEY = 'climate:anomalies:v1';
const CACHE_TTL = 10800; // 3h
const ZONE_NORMALS_KEY = 'climate:zone-normals:v1';

// Geopolitical zones (original 15)
const ZONES = [
{ name: 'Ukraine', lat: 48.4, lon: 31.2 },
{ name: 'Middle East', lat: 33.0, lon: 44.0 },
Expand All @@ -25,6 +38,19 @@ const ZONES = [
{ name: 'Caribbean', lat: 19.0, lon: -72.0 },
];

// Climate-specific zones (7 new zones)
const CLIMATE_ZONES = [
{ name: 'Arctic', lat: 70.0, lon: 0.0 },
{ name: 'Greenland', lat: 72.0, lon: -42.0 },
{ name: 'WestAntarctic', lat: -78.0, lon: -100.0 },
{ name: 'TibetanPlateau', lat: 31.0, lon: 91.0 },
{ name: 'CongoBasin', lat: -1.0, lon: 24.0 },
{ name: 'CoralTriangle', lat: -5.0, lon: 128.0 },
{ name: 'NorthAtlantic', lat: 55.0, lon: -30.0 },
];

const ALL_ZONES = [...ZONES, ...CLIMATE_ZONES];

function avg(arr) {
return arr.length ? arr.reduce((s, v) => s + v, 0) / arr.length : 0;
}
Expand All @@ -51,7 +77,37 @@ function classifyType(tempDelta, precipDelta) {
return 'ANOMALY_TYPE_COLD';
}

async function fetchZone(zone, startDate, endDate) {
/**
* Fetch zone normals from Redis cache.
* Returns a map of zone name -> { tempMean, precipMean } for the current month.
*/
async function fetchZoneNormalsFromRedis() {
const { url, token } = getRedisCredentials();
const resp = await fetch(`${url}/get/${encodeURIComponent(ZONE_NORMALS_KEY)}`, {
headers: { Authorization: `Bearer ${token}` },
signal: AbortSignal.timeout(10_000),
});

if (!resp.ok) {
console.log('[CLIMATE] Zone normals not in cache — normals seeder may not have run yet');
return null;
}

const data = await resp.json();
if (!data.result) return null;

try {
const parsed = JSON.parse(data.result);
return parsed.zones || null;
} catch {
return null;
}
}

/**
* Fetch current conditions for a zone and compare against WMO normals.
*/
async function fetchZone(zone, normals, startDate, endDate) {
const url = `https://archive-api.open-meteo.com/v1/archive?latitude=${zone.lat}&longitude=${zone.lon}&start_date=${startDate}&end_date=${endDate}&daily=temperature_2m_mean,precipitation_sum&timezone=UTC`;

const resp = await fetch(url, {
Expand All @@ -73,15 +129,55 @@ async function fetchZone(zone, startDate, endDate) {
}
}

if (temps.length < 14) return null;
if (temps.length < 7) return null;

// Use last 7 days as current period
const recentTemps = temps.slice(-7);
const baselineTemps = temps.slice(0, -7);
const recentPrecips = precips.slice(-7);
const baselinePrecips = precips.slice(0, -7);

const tempDelta = Math.round((avg(recentTemps) - avg(baselineTemps)) * 10) / 10;
const precipDelta = Math.round((avg(recentPrecips) - avg(baselinePrecips)) * 10) / 10;
const currentTempMean = avg(recentTemps);
const currentPrecipMean = avg(recentPrecips);

// Find the normal for this zone and current month
const currentMonth = new Date().getMonth() + 1; // 1-12
const zoneNormal = normals?.find((n) => n.zone === zone.name);

if (!zoneNormal) {
// Fallback: compute from previous 30 days if normals not available
// (This is the old behavior for backwards compatibility during transition)
const baselineTemps = temps.slice(0, -7);
const baselinePrecips = precips.slice(0, -7);

if (baselineTemps.length < 7) return null;

const baselineTempMean = avg(baselineTemps);
const baselinePrecipMean = avg(baselinePrecips);

const tempDelta = Math.round((currentTempMean - baselineTempMean) * 10) / 10;
const precipDelta = Math.round((currentPrecipMean - baselinePrecipMean) * 10) / 10;

return {
zone: zone.name,
location: { latitude: zone.lat, longitude: zone.lon },
tempDelta,
precipDelta,
severity: classifySeverity(tempDelta, precipDelta),
type: classifyType(tempDelta, precipDelta),
period: `${startDate} to ${endDate}`,
baselineSource: 'rolling-30d-fallback',
};
Comment on lines +145 to +168
Copy link
Copy Markdown

@greptile-apps greptile-apps bot Mar 29, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Rolling fallback always returns null — broken by date-range reduction

The date range fetched was reduced from 30 days to 7 days (startDate = Date.now() - 7 days), but the rolling fallback still tries to split the data into a baseline window + a recent window:

const baselineTemps = temps.slice(0, -7); // Empty when only 7 days fetched

With only 7 days of data available, baselineTemps is always [], so baselineTemps.length < 7 is always true, and the function always return null.

This is critical because the normals seeder (seed-climate-zone-normals.mjs) only seeds normals for 7 CLIMATE_ZONES — it does not seed the original 15 geopolitical zones (Ukraine, Middle East, Sahel, etc.). So every one of those 15 zones will hit this fallback and return null.

Net effect: At most 7 anomalies (from CLIMATE_ZONES) can be produced, but MIN_ZONES = ceil(22 × 2/3) = 15. The seeder will throw and abort on every single run after this deployment, silently preserving the old (increasingly stale) Redis data.

Two separate fixes are needed:

  1. Add all 22 ALL_ZONES to seed-climate-zone-normals.mjs, not just CLIMATE_ZONES.
  2. Fix the fallback to fetch 30 days when normals are unavailable, or explicitly return null without the broken baseline split (since the window is only 7 days).

}

// Use WMO normal for current month
const monthNormal = zoneNormal.normals?.find((n) => n.month === currentMonth);

if (!monthNormal) {
console.log(`[CLIMATE] ${zone.name}: No normal for month ${currentMonth}`);
return null;
}

const tempDelta = Math.round((currentTempMean - monthNormal.tempMean) * 10) / 10;
const precipDelta = Math.round((currentPrecipMean - monthNormal.precipMean) * 10) / 10;

return {
zone: zone.name,
Expand All @@ -91,18 +187,33 @@ async function fetchZone(zone, startDate, endDate) {
severity: classifySeverity(tempDelta, precipDelta),
type: classifyType(tempDelta, precipDelta),
period: `${startDate} to ${endDate}`,
baselineSource: 'wmo-30y-normals',
baseline: {
tempMean: monthNormal.tempMean,
precipMean: monthNormal.precipMean,
month: monthNormal.monthName,
period: zoneNormal.period,
},
};
}

async function fetchClimateAnomalies() {
const endDate = new Date().toISOString().slice(0, 10);
const startDate = new Date(Date.now() - 30 * 24 * 60 * 60 * 1000).toISOString().slice(0, 10);
const startDate = new Date(Date.now() - 7 * 24 * 60 * 60 * 1000).toISOString().slice(0, 10);

// Try to fetch WMO normals from Redis
const normals = await fetchZoneNormalsFromRedis();
if (normals) {
console.log(`[CLIMATE] Using WMO 30-year normals for ${normals.length} zones`);
} else {
console.log('[CLIMATE] Normals not available — using 30-day rolling fallback');
}

const anomalies = [];
let failures = 0;
for (const zone of ZONES) {
for (const zone of ALL_ZONES) {
try {
const result = await fetchZone(zone, startDate, endDate);
const result = await fetchZone(zone, normals, startDate, endDate);
if (result != null) anomalies.push(result);
} catch (err) {
console.log(` [CLIMATE] ${err?.message ?? err}`);
Expand All @@ -111,23 +222,24 @@ async function fetchClimateAnomalies() {
await new Promise((r) => setTimeout(r, 200));
}

const MIN_ZONES = Math.ceil(ZONES.length * 2 / 3);
const MIN_ZONES = Math.ceil(ALL_ZONES.length * 2 / 3);
if (anomalies.length < MIN_ZONES) {
throw new Error(`Only ${anomalies.length}/${ZONES.length} zones returned data (${failures} errors) — skipping write to preserve previous Redis data`);
throw new Error(`Only ${anomalies.length}/${ALL_ZONES.length} zones returned data (${failures} errors) — skipping write to preserve previous Redis data`);
}

return { anomalies, pagination: undefined };
}

function validate(data) {
return Array.isArray(data?.anomalies) && data.anomalies.length >= Math.ceil(ZONES.length * 2 / 3);
return Array.isArray(data?.anomalies) && data.anomalies.length >= Math.ceil(ALL_ZONES.length * 2 / 3);
}

runSeed('climate', 'anomalies', CANONICAL_KEY, fetchClimateAnomalies, {
validateFn: validate,
ttlSeconds: CACHE_TTL,
sourceVersion: 'open-meteo-archive-30d',
sourceVersion: 'open-meteo-archive-wmo-normals',
}).catch((err) => {
const _cause = err.cause ? ` (cause: ${err.cause.message || err.cause.code || err.cause})` : ''; console.error('FATAL:', (err.message || err) + _cause);
const _cause = err.cause ? ` (cause: ${err.cause.message || err.cause.code || err.cause})` : '';
console.error('FATAL:', (err.message || err) + _cause);
process.exit(1);
});
164 changes: 164 additions & 0 deletions scripts/seed-climate-zone-normals.mjs
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,164 @@
#!/usr/bin/env node
/**
* seed-climate-zone-normals.mjs
*
* Fetches WMO 30-year climatological normals (1991-2020) for each climate zone.
* These are used as the baseline for climate anomaly detection instead of the
* climatologically meaningless 30-day rolling window.
*
* Run: Monthly (1st of month, 03:00 UTC) via Railway cron
* Cache: climate:zone-normals:v1 (TTL 30 days)
*/

import { loadEnvFile, CHROME_UA, runSeed } from './_seed-utils.mjs';

loadEnvFile(import.meta.url);

const CANONICAL_KEY = 'climate:zone-normals:v1';
const CACHE_TTL = 30 * 24 * 60 * 60; // 30 days in seconds

// Climate-specific zones in addition to geopolitical zones in seed-climate-anomalies.mjs
const CLIMATE_ZONES = [
{ name: 'Arctic', lat: 70.0, lon: 0.0 }, // sea ice proxy
{ name: 'Greenland', lat: 72.0, lon: -42.0 }, // ice sheet melt
{ name: 'WestAntarctic', lat: -78.0, lon: -100.0 }, // Antarctic Ice Sheet
{ name: 'TibetanPlateau', lat: 31.0, lon: 91.0 }, // third pole
{ name: 'CongoBasin', lat: -1.0, lon: 24.0 }, // largest tropical forest after Amazon
{ name: 'CoralTriangle', lat: -5.0, lon: 128.0 }, // reef bleaching proxy
{ name: 'NorthAtlantic', lat: 55.0, lon: -30.0 }, // AMOC slowdown signal
];
Comment on lines +40 to +48
Copy link
Copy Markdown

@greptile-apps greptile-apps bot Mar 29, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Normals seeder only covers 7 of 22 zones — original zones never get WMO baseline

This seeder seeds normals for only the 7 new CLIMATE_ZONES, but seed-climate-anomalies.mjs uses ALL_ZONES (22 zones) and looks up normals for every one:

// In seed-climate-anomalies.mjs
const zoneNormal = normals?.find((n) => n.zone === zone.name);

The 15 original geopolitical zones (Ukraine, Middle East, Sahel, Horn of Africa, etc.) will never match a cached normal and will always fall through to the broken rolling-baseline path (see companion comment on seed-climate-anomalies.mjs).

To make the WMO baseline work for ALL zones, CLIMATE_ZONES here should be replaced with — or merged with — the full ALL_ZONES list from seed-climate-anomalies.mjs. Since both files share the same zone definitions, consider extracting them into a shared constant (e.g., a shared/climate-zones.json) to prevent future drift.


// Month names for logging
const MONTH_NAMES = ['Jan', 'Feb', 'Mar', 'Apr', 'May', 'Jun', 'Jul', 'Aug', 'Sep', 'Oct', 'Nov', 'Dec'];

/**
* Fetch monthly normals for a zone using Open-Meteo archive API.
* We fetch 1991-2020 data and aggregate to monthly means per calendar month.
*/
async function fetchZoneNormals(zone) {
const monthlyNormals = {};

// Initialize all months
for (let m = 1; m <= 12; m++) {
monthlyNormals[m] = { temps: [], precips: [] };
}

// Fetch each year in chunks to avoid overwhelming the API
// Open-Meteo supports date ranges, so we fetch entire years
const startYear = 1991;
const endYear = 2020;

for (let year = startYear; year <= endYear; year++) {
const yearStart = `${year}-01-01`;
const yearEnd = `${year}-12-31`;
const url = `https://archive-api.open-meteo.com/v1/archive?latitude=${zone.lat}&longitude=${zone.lon}&start_date=${yearStart}&end_date=${yearEnd}&daily=temperature_2m_mean,precipitation_sum&timezone=UTC`;

try {
const resp = await fetch(url, {
headers: { 'User-Agent': CHROME_UA },
signal: AbortSignal.timeout(20_000),
});

if (!resp.ok) {
console.log(` [ZONE_NORMALS] ${zone.name} ${year}: HTTP ${resp.status}`);
continue;
}

const data = await resp.json();
const dailyTemps = data.daily?.temperature_2m_mean ?? [];
const dailyPrecips = data.daily?.precipitation_sum ?? [];
const dailyDates = data.daily?.time ?? [];

// Aggregate to monthly
for (let i = 0; i < dailyDates.length; i++) {
const dateStr = dailyDates[i];
if (!dateStr) continue;
const month = parseInt(dateStr.slice(5, 7), 10);
const temp = dailyTemps[i];
const precip = dailyPrecips[i];

if (temp != null && precip != null) {
monthlyNormals[month].temps.push(temp);
monthlyNormals[month].precips.push(precip);
}
}
} catch (err) {
console.log(` [ZONE_NORMALS] ${zone.name} ${year}: ${err?.message ?? err}`);
}

// Rate limit to be nice to Open-Meteo
await new Promise((r) => setTimeout(r, 100));
}

// Compute monthly means
const normals = [];
for (let month = 1; month <= 12; month++) {
const { temps, precips } = monthlyNormals[month];
if (temps.length === 0) {
console.log(` [ZONE_NORMALS] ${zone.name} ${MONTH_NAMES[month - 1]}: No data`);
continue;
}

const avgTemp = temps.reduce((s, v) => s + v, 0) / temps.length;
const avgPrecip = precips.reduce((s, v) => s + v, 0) / precips.length;

normals.push({
month,
monthName: MONTH_NAMES[month - 1],
tempMean: Math.round(avgTemp * 100) / 100,
precipMean: Math.round(avgPrecip * 100) / 100,
sampleCount: temps.length,
});
}

return {
zone: zone.name,
location: { latitude: zone.lat, longitude: zone.lon },
normals,
period: '1991-2020',
computedAt: new Date().toISOString(),
};
}

async function fetchAllZoneNormals() {
const allNormals = [];
let failures = 0;

for (const zone of CLIMATE_ZONES) {
console.log(`[ZONE_NORMALS] Fetching ${zone.name} (${zone.lat}, ${zone.lon})...`);
try {
const result = await fetchZoneNormals(zone);
if (result && result.normals.length > 0) {
allNormals.push(result);
console.log(` → ${result.normals.length} months, ${result.normals[0].sampleCount}+ samples/month`);
} else {
failures++;
}
} catch (err) {
console.log(` [ZONE_NORMALS] ${zone.name}: ${err?.message ?? err}`);
failures++;
}
}

if (allNormals.length === 0) {
throw new Error(`No zone normals fetched (${failures} failures)`);
}

console.log(`[ZONE_NORMALS] Completed: ${allNormals.length}/${CLIMATE_ZONES.length} zones`);

return { zones: allNormals, pagination: undefined };
}

function validate(data) {
return Array.isArray(data?.zones) && data.zones.length > 0;
}

runSeed('climate', 'zone-normals', CANONICAL_KEY, fetchAllZoneNormals, {
validateFn: validate,
ttlSeconds: CACHE_TTL,
sourceVersion: 'open-meteo-archive-wmo-normals',
}).catch((err) => {
const _cause = err.cause ? ` (cause: ${err.cause.message || err.cause.code || err.cause})` : '';
console.error('FATAL:', (err.message || err) + _cause);
process.exit(1);
});
3 changes: 2 additions & 1 deletion server/_shared/cache-keys.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,8 @@ export const BOOTSTRAP_CACHE_KEYS: Record<string, string> = {
chokepointTransits: 'supply_chain:chokepoint_transits:v1',
minerals: 'supply_chain:minerals:v2',
giving: 'giving:summary:v1',
climateAnomalies: 'climate:anomalies:v1',
climateAnomalies: 'climate:anomalies:v1',
climateZoneNormals: 'climate:zone-normals:v1',
Copy link
Copy Markdown

@greptile-apps greptile-apps bot Mar 29, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 climateZoneNormals registered in BOOTSTRAP_CACHE_KEYS but not wired in api/bootstrap.js or BOOTSTRAP_TIERS

climateZoneNormals was added to server/_shared/cache-keys.ts but is absent from both:

  • api/bootstrap.js (the actual Edge Function that serves bootstrap data) — which maintains its own hardcoded copy of the key registry and is the file that AGENTS.md requires be updated for new data sources
  • BOOTSTRAP_TIERS in this same file (line ~82) — which governs cache tier assignment

If climateZoneNormals is intended as frontend-accessible bootstrap data, both api/bootstrap.js and BOOTSTRAP_TIERS need to be updated. If it's purely an internal seed-pipeline artifact (the anomaly seeder reads it, but the UI doesn't), it arguably shouldn't be in BOOTSTRAP_CACHE_KEYS at all, which is documented as being for the bootstrap endpoint.

radiationWatch: 'radiation:observations:v1',
thermalEscalation: 'thermal:escalation:v1',
crossSourceSignals: 'intelligence:cross-source-signals:v1',
Expand Down
Loading
Loading