Add climate news seed and ListClimateNews RPC#2532
Conversation
|
Preview deployment for your docs. Learn more about Mintlify Previews.
|
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Greptile SummaryThis PR wires a full climate news intelligence pipeline: 9 RSS feeds are aggregated every 30 minutes by a new Key findings:
Confidence Score: 4/5Safe to merge after fixing the One P1 issue:
Important Files Changed
Sequence DiagramsequenceDiagram
participant Relay as ais-relay.cjs (Railway)
participant Seed as seed-climate-news.mjs
participant RSS as RSS Feeds (x9)
participant Redis as Upstash Redis
participant Edge as Vercel Edge Function
participant Client as Browser Client
loop Every 30 minutes
Relay->>Seed: execFile(node, seed-climate-news.mjs)
Seed->>RSS: fetch() x9 feeds (15s timeout each)
RSS-->>Seed: XML responses
Seed->>Seed: parseRssItems() / dedup / sort
Seed->>Redis: SET climate:news-intelligence:v1 (TTL 1800s)
Seed->>Redis: SET seed-meta:climate:news-intelligence
Seed-->>Relay: exit 0
end
Client->>Edge: GET /api/climate/v1/list-climate-news
Edge->>Redis: getCachedJson(climate:news-intelligence:v1)
Redis-->>Edge: { items[], fetchedAt }
Edge-->>Client: ListClimateNewsResponse (JSON)
|
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
scripts/seed-climate-news.mjs
Outdated
| if (items.length === 0) { | ||
| const entryRe = /<entry\b[^>]*>([\s\S]*?)<\/entry>/gi; | ||
| while ((match = entryRe.exec(bounded)) !== null) { | ||
| const block = match[1]; | ||
| const title = decodeHtmlEntities(extractTag(block, 'title')); | ||
| const url = extractLink(block); | ||
| const publishedAt = parseDateMs(block); | ||
| const rawSummary = extractTag(block, 'summary') || extractTag(block, 'content'); | ||
| if (!title || !url || !publishedAt) continue; | ||
| items.push({ | ||
| id: `${stableHash(url)}-${publishedAt}`, | ||
| title, | ||
| url, | ||
| sourceName, | ||
| publishedAt, | ||
| summary: cleanSummary(rawSummary), | ||
| }); | ||
| } | ||
| } |
There was a problem hiding this comment.
<item> blocks return zero items across the entire feed
The Atom <entry> fallback is gated on if (items.length === 0). This means it is only attempted when the entire feed contained no <item> elements. For feeds that publish a mix of both (unusual but valid), Atom entries would be silently dropped. More practically, all 9 feeds in FEEDS are standard RSS feeds, so this isn't an immediate bug — but any future feed addition that uses Atom will silently return 0 items unless this gate is removed.
Consider restructuring to always attempt both parsers and merge results (deduplicating on id), or at minimum add a comment explaining that the fallback is per-feed, not per-batch.
| function extractTag(block, tagName) { | ||
| const re = new RegExp(`<${tagName}[^>]*>(?:<!\\[CDATA\\[)?([\\s\\S]*?)(?:\\]\\]>)?<\\/${tagName}>`, 'i'); | ||
| return (block.match(re) || [])[1]?.trim() || ''; | ||
| } |
There was a problem hiding this comment.
extractTag regex mishandles CDATA sections with embedded ]]> content
The current regex makes the CDATA start/end markers optional via (?:..)?:
<${tagName}[^>]*>(?:<!\[CDATA\[)?([\s\S]*?)(?:\]\]>)?<\/${tagName}>
Because both the opening <![CDATA[ and closing ]]> markers are optional and independent, a tag such as <title><![CDATA[Breaking news ]]> & more]]></title> will capture only Breaking news (stopping at the first ]]>), silently truncating the title. The fix is to use two separate branches — one for CDATA content and one for plain text — rather than making both markers optional.
|
p1 overclaims impact. |
|
@FayezBast — good pipeline addition. A few things before merging:
Otherwise the pipeline design is solid. |
2 participants
Summary
Adds
seed-climate-news.mjsto aggregate 9 authoritative climate/environment RSS feeds intoclimate:news-intelligence:v1, wires a 30-minute relay seed loop, and exposes the data through a newListClimateNewsclimate proto RPC and server handler.Not included in this PR:
get_climate_dataexpansion to includeclimate:news-intelligence:v1Fixes #2469
Fixes #2560
Type of change
Affected areas
/api/*)