Skip to content

feat(simulation): confidence-weighted adjustments + simulationSignal trace lane#2515

Merged
koala73 merged 5 commits intomainfrom
feat/simulation-confidence-lane
Mar 29, 2026
Merged

feat(simulation): confidence-weighted adjustments + simulationSignal trace lane#2515
koala73 merged 5 commits intomainfrom
feat/simulation-confidence-lane

Conversation

@koala73
Copy link
Copy Markdown
Owner

@koala73 koala73 commented Mar 29, 2026

Summary

  • Confidence-weighted bonuses: +0.08 and +0.04 simulation bonuses now scale by simPathConfidence from SimulationTopPath.confidence. Missing/null/zero confidence falls back to 1.0 (conservative — old artifacts not penalized). Negative adjustments (-0.12/-0.15) remain flat (structural, not sim-confidence-dependent).
  • simulationSignal trace lane: applySimulationMerge attaches a compact simulationSignal object (backed, adjustmentDelta, channelSource, demoted, simPathConfidence) to each ExpandedPath when adjustment ≠ 0. Written to R2 trace artifacts via existing writeForecastTraceArtifacts.
  • simPathConfidence observability: Added simPathConfidence field to SimulationAdjustmentDetail so adjustments per path are fully inspectable in trace output.
  • Named-actor gap documented: Macro-financial theaters with role-based stateSummary.actors ("Commodity traders") have actorOverlapCount=0 by design — only named geo-political actor theaters (maritime/security) fire the +0.04 bonus.

Closes out the last two "Next (planned)" items from docs/internal/wm-mirofish-gap.md.

ForecastPanel UI chip deferred: simulationSignal lives in R2 trace artifacts. Surfacing it as a UI chip requires a proto field addition + buf generate + prediction-to-path plumbing. Tracked for a follow-up PR.

Testing

  • T-N1..T-N8: confidence weighting unit tests (full 1.0, partial 0.72 with overlap, 0.35 without, undefined fallback, zero fallback, flat negatives, simPathConfidence observability, unweighted negatives)
  • T-O1..T-O4: simulationSignal tests (backed + not-demoted, absent on zero adjustment, demoted on threshold crossing, simPathConfidence propagated)
  • 267 passing, 0 failing

Post-Deploy Monitoring & Validation

  • What to monitor: R2 trace artifacts for simulation-outcome-*.json — check simulationSignal fields on paths with non-zero simulationAdjustment
  • Expected healthy behavior: Paths with bucketChannelMatch=true have simulationSignal.backed=true, adjustmentDelta matches simulationAdjustment, simPathConfidence < 1.0 when sim output includes fractional confidence
  • Failure signal: simulationSignal absent on a path that has non-zero simulationAdjustment (would indicate applySimulationMerge early-exit regression)
  • No production runtime impact — no Redis writes or API surface changes; all changes are in trace artifact content

Compound Engineering v2.49.0
🤖 Generated with Claude Sonnet 4.6 (200K context) via Claude Code

…trace lane

Scale +0.08 and +0.04 simulation bonuses by simPathConfidence (missing/zero
falls back to 1.0 so old artifacts are not penalized). Negative adjustments
(-0.12/-0.15) remain flat — they are structural, not sim-confidence-dependent.

Attach a compact simulationSignal object (backed, adjustmentDelta, channelSource,
demoted, simPathConfidence) to each ExpandedPath when adjustment != 0. Written
to R2 trace artifacts. ForecastPanel UI chip deferred to follow-up PR (requires
proto field + buf generate + prediction-to-path plumbing).

Add simPathConfidence to SimulationAdjustmentDetail for observability.
Add T-N1..T-N8 (confidence weighting) and T-O1..T-O4 (simulationSignal) tests.

🤖 Generated with Claude Sonnet 4.6 via Claude Code (https://claude.ai/claude-code) + Compound Engineering v2.49.0

Co-Authored-By: Claude Sonnet 4.6 (200K context) <noreply@anthropic.com>
@vercel
Copy link
Copy Markdown

vercel bot commented Mar 29, 2026

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
worldmonitor Ready Ready Preview, Comment Mar 29, 2026 6:04pm

Request Review

@greptile-apps
Copy link
Copy Markdown

greptile-apps bot commented Mar 29, 2026

Greptile Summary

This PR introduces two tightly scoped improvements to the simulation merge pipeline: confidence-weighted bonuses (scaling +0.08/+0.04 by simPathConfidence from the matched SimulationTopPath) and a compact simulationSignal trace object written to each ExpandedPath when a non-zero adjustment is applied. Negative adjustments (-0.12/-0.15) intentionally remain flat. The changes are purely trace/observability additions with no Redis or API surface impact.

  • Confidence weightingsimConf is derived from bucketChannelMatch.confidence with a safe fallback: any missing/null/zero/non-finite value collapses to 1.0, preserving backward-compat with older LLM artifacts.
  • simulationSignal attachment — written by applySimulationMerge only when adjustment !== 0; carries backed, adjustmentDelta, channelSource, demoted, and simPathConfidence for downstream trace inspection.
  • SimulationAdjustmentDetail.simPathConfidence — extends the existing detail record for full per-path observability in R2 trace artifacts.
  • Tests (T-N1..T-N8, T-O1..T-O4) — comprehensive coverage of weighting, all fallback branches, flat negatives, and signal lifecycle including the zero-adjustment absent-signal case.
  • Minor JSDoc mismatch — the SimulationSignal.backed comment describes it as indicating "non-zero adjustment applied", but the implementation sets it only for adjustment > 0; negative-adjustment paths will have backed=false. The comment should be updated to reflect the actual semantics.

Confidence Score: 5/5

Safe to merge — no API surface or Redis changes; only minor JSDoc inaccuracies remain.

All findings are P2 (style/documentation). The logic for confidence weighting, fallbacks, flat negatives, and signal lifecycle is correct and thoroughly tested. The remaining JSDoc comment on SimulationSignal.backed is the only open item and does not affect runtime behavior.

scripts/seed-forecasts.types.d.ts — SimulationSignal.backed JSDoc needs a wording update to match adjustment > 0 semantics.

Important Files Changed

Filename Overview
scripts/seed-forecasts.mjs Core logic changes: confidence-weighted bonuses (+0.08/+0.04 × simConf) replace flat bonuses, and simulationSignal trace object is attached to each adjusted path; fallback logic (undefined/null/zero → 1.0) and negative-flat behavior are correct.
scripts/seed-forecasts.types.d.ts New SimulationSignal interface and additions to ExpandedPath/SimulationAdjustmentDetail; the backed field JSDoc comment is misleading (says "non-zero adjustment applied" but the impl only sets it true for adjustment > 0).
tests/forecast-trace-export.test.mjs T-N1..T-N8 and T-O1..T-O4 cover confidence weighting, fallback paths, flat negatives, and simulationSignal lifecycle; all assertions are logically consistent with the implementation.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[applySimulationMerge called] --> B{theaterResults present?}
    B -- No --> Z[return null simulationEvidence]
    B -- Yes --> C[for each ExpandedPath]
    C --> D{simResult found for path?}
    D -- No --> C
    D -- Yes --> E[computeSimulationAdjustment]
    E --> F{bucketChannelMatch?}
    F -- Yes --> G[rawConf = match.confidence]
    G --> H{isFinite and > 0?}
    H -- No --> I[simConf = 1.0 fallback]
    H -- Yes --> J[simConf = min 1 rawConf]
    I & J --> K[adjustment += 0.08 x simConf]
    K --> L{actorOverlap >= 2?}
    L -- Yes --> M[adjustment += 0.04 x simConf]
    L -- No --> N
    M --> N{invalidator hit?}
    F -- No --> N
    N -- Yes --> O[adjustment -= 0.12 flat]
    N -- No --> P{stabilizer hit?}
    O --> P
    P -- Yes --> Q[adjustment -= 0.15 flat]
    P -- No --> R[return adjustment + details]
    Q --> R
    R --> S{adjustment == 0?}
    S -- Yes --> C
    S -- No --> T[write simulationSignal to path]
    T --> U{wasAccepted and score < threshold?}
    U -- Yes --> V[path.demotedBySimulation = true]
    U -- No --> W{was rejected and score >= threshold?}
    W -- Yes --> X[path.promotedBySimulation = true]
    W -- No --> C
    V & X --> C
Loading

Reviews (1): Last reviewed commit: "feat(simulation): confidence-weighted ad..." | Re-trigger Greptile

* Written by applySimulationMerge; rendered as a chip in ForecastPanel.
*/
interface SimulationSignal {
/** Non-zero simulation adjustment was applied (positive = promoted, negative = weakened). */
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 backed JSDoc contradicts the implementation

The comment reads "Non-zero simulation adjustment was applied (positive = promoted, negative = weakened)", which implies backed would be true for any non-zero adjustment, including negative ones. But the implementation in applySimulationMerge sets backed: adjustment > 0, so paths that receive a negative adjustment (e.g. an invalidator hit) will have backed=false even though the signal is present and has a non-zero adjustmentDelta.

A future consumer checking simulationSignal.backed to determine "did simulation touch this path?" would get a false negative for all weakened paths. Aligning the comment with the actual semantics avoids that confusion:

Suggested change
/** Non-zero simulation adjustment was applied (positive = promoted, negative = weakened). */
/** True when adjustment > 0 — simulation positively supports this path (promoted or reinforced). False for paths weakened by simulation. */
backed: boolean;

Comment on lines +11454 to +11464
adjustment += +parseFloat((0.08 * simConf).toFixed(3));
details.bucketChannelMatch = true;
details.simPathConfidence = simConf;
const simActors = new Set((Array.isArray(bucketChannelMatch.keyActors) ? bucketChannelMatch.keyActors : []).map(normalizeActorName));
const overlap = candidateActors.filter((a) => simActors.has(a));
details.actorOverlapCount = overlap.length;
// Overlap bonus fires only when both sides have named geo-political actors.
// Macro-financial theaters with role-based stateSummary.actors (e.g. "Commodity traders",
// "Central banks") will have actorOverlapCount=0 — this is expected, not a bug.
if (overlap.length >= 2) {
adjustment += 0.04;
adjustment += +parseFloat((0.04 * simConf).toFixed(3));
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Redundant unary + on parseFloat return value

parseFloat(...) already returns a number (or NaN), so the leading + cast is a no-op in both places. It doesn't affect correctness, but it adds visual noise and suggests the author may have intended Number(...) or +(str) on a string that wasn't already numeric. Consider removing the outer + for clarity:

Suggested change
adjustment += +parseFloat((0.08 * simConf).toFixed(3));
details.bucketChannelMatch = true;
details.simPathConfidence = simConf;
const simActors = new Set((Array.isArray(bucketChannelMatch.keyActors) ? bucketChannelMatch.keyActors : []).map(normalizeActorName));
const overlap = candidateActors.filter((a) => simActors.has(a));
details.actorOverlapCount = overlap.length;
// Overlap bonus fires only when both sides have named geo-political actors.
// Macro-financial theaters with role-based stateSummary.actors (e.g. "Commodity traders",
// "Central banks") will have actorOverlapCount=0 — this is expected, not a bug.
if (overlap.length >= 2) {
adjustment += 0.04;
adjustment += +parseFloat((0.04 * simConf).toFixed(3));
adjustment += parseFloat((0.08 * simConf).toFixed(3));
details.bucketChannelMatch = true;
details.simPathConfidence = simConf;
const simActors = new Set((Array.isArray(bucketChannelMatch.keyActors) ? bucketChannelMatch.keyActors : []).map(normalizeActorName));
const overlap = candidateActors.filter((a) => simActors.has(a));
details.actorOverlapCount = overlap.length;
// Overlap bonus fires only when both sides have named geo-political actors.
// Macro-financial theaters with role-based stateSummary.actors (e.g. "Commodity traders",
// "Central banks") will have actorOverlapCount=0 — this is expected, not a bug.
if (overlap.length >= 2) {
adjustment += parseFloat((0.04 * simConf).toFixed(3));
}

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

…gnal to scorecard summaries

P1: explicit confidence=0 from LLM now correctly yields simConf=0 (no positive bonus)
instead of falling back to 1.0. Absent/non-finite confidence still uses 1.0 fallback
(conservative — old LLM artifacts without the field are not penalized). The previous
rawConf > 0 guard conflated "absent" and "explicitly unsupported" paths.

P2: summarizeImpactPathScore now forwards simulationSignal into scorecard summaries,
so path-scorecards.json and impact-expansion-debug.json include the new lane alongside
simulationAdjustment and mergedAcceptanceScore. Only forecast-eval.json had it before.

Also exports summarizeImpactPathScore for direct unit testing.

Tests: T-N5 corrected (explicit 0 → no bonus), T-N5b (zero conf + invalidator still
fires flat -0.12), T-O5 (summarizeImpactPathScore includes simulationSignal),
T-O6 (omits field when absent). 270 passing.

🤖 Generated with Claude Sonnet 4.6 via Claude Code (https://claude.ai/claude-code) + Compound Engineering v2.49.0

Co-Authored-By: Claude Sonnet 4.6 (200K context) <noreply@anthropic.com>
koala73 and others added 3 commits March 29, 2026 21:50
…Confidence comment

P1 (blocking): applySimulationMerge now deletes simulationAdjustment, mergedAcceptanceScore,
simulationSignal, demotedBySimulation, and promotedBySimulation from every expanded path
before running computeSimulationAdjustment. This fires before any early-continue
(no theater match, no candidate packet, zero confidence), so reloaded forecast-eval.json
paths from applyPostSimulationRescore never retain stale simulation metadata from a prior cycle.

P3: corrected simPathConfidence JSDoc in SimulationAdjustmentDetail and SimulationSignal —
absent/non-finite → 1.0 fallback, explicit 0 preserved as 0 (was: "missing/null/zero fall back to 1.0").

Tests: T-O7 (zero-confidence match clears stale fields), T-O8 (no-theater match clears
stale fields). 272 passing.

🤖 Generated with Claude Sonnet 4.6 via Claude Code (https://claude.ai/claude-code) + Compound Engineering v2.49.0

Co-Authored-By: Claude Sonnet 4.6 (200K context) <noreply@anthropic.com>
@koala73 koala73 merged commit 5bfd7bc into main Mar 29, 2026
6 of 7 checks passed
@koala73 koala73 deleted the feat/simulation-confidence-lane branch March 29, 2026 18:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant