chore(railway): sync regulatory seed cron schedule#2569
chore(railway): sync regulatory seed cron schedule#2569lspassos1 wants to merge 12 commits intokoala73:mainfrom
Conversation
Add a standalone seeder that fetches and normalizes SEC, CFTC, Federal Reserve, FDIC, and FINRA regulatory feeds without introducing new dependencies. The script stays import-safe, tolerates partial feed failure, and emits JSON for the fetch/parse-only phase of the pipeline. Unit tests cover RSS/Atom parsing, deduplication, ordering, and degraded-feed behavior. Refs koala73#2492 Refs koala73#2493 Refs koala73#2494 Refs koala73#2495
Build on the standalone RSS fetcher by adding keyword-based tier classification, aggregate payload counts, and runSeed integration for regulatory:actions:v1. The updated tests cover matched keywords, payload stats, and the runSeed wiring needed for Redis publication. Refs koala73#2493 Depends on koala73#2564
Add regulatory:actions:v1 as a new cross-source input, map regulatory actions into the policy category, and emit CROSS_SOURCE_SIGNAL_TYPE_REGULATORY_ACTION signals for recent high/medium items. The new test covers severity scoring and composite escalation when policy, financial, and economic signals co-fire in Global Markets. Refs koala73#2494 Depends on koala73#2567
Extend the Railway seed-service sync script to enforce the expected cronSchedule for seed-regulatory-actions, while continuing to validate watch patterns and start commands. Add a focused test for the new cronSchedule path and fail fast when the required seed-regulatory-actions service is missing. Refs koala73#2495 Depends on koala73#2568
|
@lspassos1 is attempting to deploy a commit to the Elie Team on Vercel. A member of the Team first needs to authorize it. |
Greptile SummaryThis PR introduces a new Key findings:
Confidence Score: 4/5
Important Files Changed
Sequence DiagramsequenceDiagram
participant Cron as Railway Cron<br/>(0 */2 * * *)
participant Seed as seed-regulatory-actions
participant Feeds as RSS Feeds<br/>(SEC, CFTC, Fed, FDIC, FINRA)
participant Redis as Redis
participant CSS as seed-cross-source-signals
participant Bootstrap as api/bootstrap.js
participant Client as Dashboard Client
Cron->>Seed: trigger every 2h
Seed->>Feeds: fetch RSS/Atom (parallel, 15s timeout)
Feeds-->>Seed: XML responses
Seed->>Seed: parse → normalize → dedupe → classify (high/medium/low)
Seed->>Redis: SET regulatory:actions:v1 (TTL 7200s)
Seed->>Redis: SET seed-meta:regulatory:actions
Cron->>CSS: trigger (separate schedule)
CSS->>Redis: GET regulatory:actions:v1 + other source keys
Redis-->>CSS: payload
CSS->>CSS: extractRegulatoryAction (48h cutoff, top-3 non-low)
CSS->>Redis: SET intelligence:cross-source-signals:v1
Client->>Bootstrap: GET /api/bootstrap
Bootstrap->>Redis: MGET [...keys] ⚠️ regulatory:actions:v1 missing
Redis-->>Bootstrap: bulk data (no regulatory actions)
Bootstrap-->>Client: hydration payload (regulatory gap)
Reviews (1): Last reviewed commit: "chore(railway): sync regulatory seed cro..." | Re-trigger Greptile |
|
Applied this live in Railway.
Refs #2495. |
Use the repository-standard fetch wrapper in the seeder defaults, keep the documented FINRA HTTP exception in place, and include publish time in generated action ids to avoid same-day collisions. Validated with: node --test tests/regulatory-seed-unit.test.mjs; node scripts/seed-regulatory-actions.mjs | head -n 20
Clean up the leftover cherry-pick marker after carrying the shared seeder hardening changes onto this branch. Validated with: node --test tests/regulatory-seed-unit.test.mjs and a local fetchRegulatoryActionPayload smoke check.
Explicitly sort regulatory actions by publishedAt inside the cross-source extractor before applying the 3-item limit, and cover the behavior with an out-of-order payload test. Validated with: node --test tests/regulatory-seed-unit.test.mjs and node --test tests/cross-source-signals-regulatory.test.mjs.
Compare Railway watchPatterns using a copied array so the validation path stays side-effect free. Validated with: node --test tests/regulatory-seed-unit.test.mjs; node --test tests/cross-source-signals-regulatory.test.mjs; node --test tests/railway-set-watch-paths.test.mjs; node --test tests/bootstrap.test.mjs.
|
Follow-up on the bootstrap thread: |
Review — PR #2569 (Railway cron schedule)Why this PR? Configures Railway cron for the regulatory seed and includes hardening fixes from the chain. Blocking1. TTL = cron interval (reiterated from #2567). 2. Missing regulatoryActions: { key: 'seed-meta:regulatory:actions', maxStaleMin: 240 },3. Missing STANDALONE_KEYS registration in regulatoryActions: 'regulatory:actions:v1',Suggestions
Across the chainThe code quality is strong. @lspassos1 clearly studied the project patterns (deferred fetch lambda,
Once these are addressed, the chain is merge-ready. Happy to re-review. |
Extract RSS and Atom descriptions into the normalized action payload so later classifier work can use the same parsed feed output. Also adds @ts-check and documents the FINRA HTTP feed constraint.
Raise the Redis retention window, classify against combined title and description text, reserve low for routine notices, and export the shared regulatory cache key for downstream health wiring.
Reject malformed timestamps, sort regulatory actions by tier before recency, and keep only high/medium signals in the cross-source extractor.
Register the regulatory actions Redis key and its seed-meta freshness window in the health endpoint without expanding bootstrap coverage.
|
@koala73 Thanks for the review. I applied the requested changes across the chain: description parsing in the feed payload, the TTL/cache-key/health wiring, the classifier updates, and the cross-source selection fixes. When you have a moment, could you take another look? |
Summary
This extends
scripts/railway-set-watch-paths.mjsso the seed-service sync also enforces the expected RailwaycronScheduleforseed-regulatory-actions, alongside the existingwatchPatternsandstartCommandchecks.Root cause
The repository had no codified check for the regulatory seeder's 2-hour cron cadence, so the Railway scheduler state for
seed-regulatory-actionscould drift from the intended7200sRedis TTL contract.Changes
seed-regulatory-actionsservice to exist before applying settingscronSchedulethroughserviceInstanceUpdate0 */2 * * *tests/railway-set-watch-paths.test.mjscovering the required service, expected schedule, andcronScheduleupdate pathValidation
node --test tests/railway-set-watch-paths.test.mjsnode scripts/railway-set-watch-paths.mjs --dry-run-> fails locally withNo Railway token found. Set RAILWAY_TOKEN or run \railway login`.`Risk
Low code risk. The script change is narrow and test-covered, but the live Railway schedule was not applied from this environment because no Railway auth is available.
Note
This branch is currently stacked on #2568, which depends on #2567 and #2564. Until those PRs merge, this PR includes the parent commits in the diff.
Depends on #2568
Refs #2495