feat(simulation): confidence-weighted adjustments + simulationSignal trace lane#2515
feat(simulation): confidence-weighted adjustments + simulationSignal trace lane#2515
Conversation
…trace lane Scale +0.08 and +0.04 simulation bonuses by simPathConfidence (missing/zero falls back to 1.0 so old artifacts are not penalized). Negative adjustments (-0.12/-0.15) remain flat — they are structural, not sim-confidence-dependent. Attach a compact simulationSignal object (backed, adjustmentDelta, channelSource, demoted, simPathConfidence) to each ExpandedPath when adjustment != 0. Written to R2 trace artifacts. ForecastPanel UI chip deferred to follow-up PR (requires proto field + buf generate + prediction-to-path plumbing). Add simPathConfidence to SimulationAdjustmentDetail for observability. Add T-N1..T-N8 (confidence weighting) and T-O1..T-O4 (simulationSignal) tests. 🤖 Generated with Claude Sonnet 4.6 via Claude Code (https://claude.ai/claude-code) + Compound Engineering v2.49.0 Co-Authored-By: Claude Sonnet 4.6 (200K context) <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Greptile SummaryThis PR introduces two tightly scoped improvements to the simulation merge pipeline: confidence-weighted bonuses (scaling
Confidence Score: 5/5Safe to merge — no API surface or Redis changes; only minor JSDoc inaccuracies remain. All findings are P2 (style/documentation). The logic for confidence weighting, fallbacks, flat negatives, and signal lifecycle is correct and thoroughly tested. The remaining JSDoc comment on scripts/seed-forecasts.types.d.ts — Important Files Changed
Flowchart%%{init: {'theme': 'neutral'}}%%
flowchart TD
A[applySimulationMerge called] --> B{theaterResults present?}
B -- No --> Z[return null simulationEvidence]
B -- Yes --> C[for each ExpandedPath]
C --> D{simResult found for path?}
D -- No --> C
D -- Yes --> E[computeSimulationAdjustment]
E --> F{bucketChannelMatch?}
F -- Yes --> G[rawConf = match.confidence]
G --> H{isFinite and > 0?}
H -- No --> I[simConf = 1.0 fallback]
H -- Yes --> J[simConf = min 1 rawConf]
I & J --> K[adjustment += 0.08 x simConf]
K --> L{actorOverlap >= 2?}
L -- Yes --> M[adjustment += 0.04 x simConf]
L -- No --> N
M --> N{invalidator hit?}
F -- No --> N
N -- Yes --> O[adjustment -= 0.12 flat]
N -- No --> P{stabilizer hit?}
O --> P
P -- Yes --> Q[adjustment -= 0.15 flat]
P -- No --> R[return adjustment + details]
Q --> R
R --> S{adjustment == 0?}
S -- Yes --> C
S -- No --> T[write simulationSignal to path]
T --> U{wasAccepted and score < threshold?}
U -- Yes --> V[path.demotedBySimulation = true]
U -- No --> W{was rejected and score >= threshold?}
W -- Yes --> X[path.promotedBySimulation = true]
W -- No --> C
V & X --> C
Reviews (1): Last reviewed commit: "feat(simulation): confidence-weighted ad..." | Re-trigger Greptile |
scripts/seed-forecasts.types.d.ts
Outdated
| * Written by applySimulationMerge; rendered as a chip in ForecastPanel. | ||
| */ | ||
| interface SimulationSignal { | ||
| /** Non-zero simulation adjustment was applied (positive = promoted, negative = weakened). */ |
There was a problem hiding this comment.
backed JSDoc contradicts the implementation
The comment reads "Non-zero simulation adjustment was applied (positive = promoted, negative = weakened)", which implies backed would be true for any non-zero adjustment, including negative ones. But the implementation in applySimulationMerge sets backed: adjustment > 0, so paths that receive a negative adjustment (e.g. an invalidator hit) will have backed=false even though the signal is present and has a non-zero adjustmentDelta.
A future consumer checking simulationSignal.backed to determine "did simulation touch this path?" would get a false negative for all weakened paths. Aligning the comment with the actual semantics avoids that confusion:
| /** Non-zero simulation adjustment was applied (positive = promoted, negative = weakened). */ | |
| /** True when adjustment > 0 — simulation positively supports this path (promoted or reinforced). False for paths weakened by simulation. */ | |
| backed: boolean; |
| adjustment += +parseFloat((0.08 * simConf).toFixed(3)); | ||
| details.bucketChannelMatch = true; | ||
| details.simPathConfidence = simConf; | ||
| const simActors = new Set((Array.isArray(bucketChannelMatch.keyActors) ? bucketChannelMatch.keyActors : []).map(normalizeActorName)); | ||
| const overlap = candidateActors.filter((a) => simActors.has(a)); | ||
| details.actorOverlapCount = overlap.length; | ||
| // Overlap bonus fires only when both sides have named geo-political actors. | ||
| // Macro-financial theaters with role-based stateSummary.actors (e.g. "Commodity traders", | ||
| // "Central banks") will have actorOverlapCount=0 — this is expected, not a bug. | ||
| if (overlap.length >= 2) { | ||
| adjustment += 0.04; | ||
| adjustment += +parseFloat((0.04 * simConf).toFixed(3)); |
There was a problem hiding this comment.
Redundant unary
+ on parseFloat return value
parseFloat(...) already returns a number (or NaN), so the leading + cast is a no-op in both places. It doesn't affect correctness, but it adds visual noise and suggests the author may have intended Number(...) or +(str) on a string that wasn't already numeric. Consider removing the outer + for clarity:
| adjustment += +parseFloat((0.08 * simConf).toFixed(3)); | |
| details.bucketChannelMatch = true; | |
| details.simPathConfidence = simConf; | |
| const simActors = new Set((Array.isArray(bucketChannelMatch.keyActors) ? bucketChannelMatch.keyActors : []).map(normalizeActorName)); | |
| const overlap = candidateActors.filter((a) => simActors.has(a)); | |
| details.actorOverlapCount = overlap.length; | |
| // Overlap bonus fires only when both sides have named geo-political actors. | |
| // Macro-financial theaters with role-based stateSummary.actors (e.g. "Commodity traders", | |
| // "Central banks") will have actorOverlapCount=0 — this is expected, not a bug. | |
| if (overlap.length >= 2) { | |
| adjustment += 0.04; | |
| adjustment += +parseFloat((0.04 * simConf).toFixed(3)); | |
| adjustment += parseFloat((0.08 * simConf).toFixed(3)); | |
| details.bucketChannelMatch = true; | |
| details.simPathConfidence = simConf; | |
| const simActors = new Set((Array.isArray(bucketChannelMatch.keyActors) ? bucketChannelMatch.keyActors : []).map(normalizeActorName)); | |
| const overlap = candidateActors.filter((a) => simActors.has(a)); | |
| details.actorOverlapCount = overlap.length; | |
| // Overlap bonus fires only when both sides have named geo-political actors. | |
| // Macro-financial theaters with role-based stateSummary.actors (e.g. "Commodity traders", | |
| // "Central banks") will have actorOverlapCount=0 — this is expected, not a bug. | |
| if (overlap.length >= 2) { | |
| adjustment += parseFloat((0.04 * simConf).toFixed(3)); | |
| } |
Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!
…gnal to scorecard summaries P1: explicit confidence=0 from LLM now correctly yields simConf=0 (no positive bonus) instead of falling back to 1.0. Absent/non-finite confidence still uses 1.0 fallback (conservative — old LLM artifacts without the field are not penalized). The previous rawConf > 0 guard conflated "absent" and "explicitly unsupported" paths. P2: summarizeImpactPathScore now forwards simulationSignal into scorecard summaries, so path-scorecards.json and impact-expansion-debug.json include the new lane alongside simulationAdjustment and mergedAcceptanceScore. Only forecast-eval.json had it before. Also exports summarizeImpactPathScore for direct unit testing. Tests: T-N5 corrected (explicit 0 → no bonus), T-N5b (zero conf + invalidator still fires flat -0.12), T-O5 (summarizeImpactPathScore includes simulationSignal), T-O6 (omits field when absent). 270 passing. 🤖 Generated with Claude Sonnet 4.6 via Claude Code (https://claude.ai/claude-code) + Compound Engineering v2.49.0 Co-Authored-By: Claude Sonnet 4.6 (200K context) <noreply@anthropic.com>
…Confidence comment P1 (blocking): applySimulationMerge now deletes simulationAdjustment, mergedAcceptanceScore, simulationSignal, demotedBySimulation, and promotedBySimulation from every expanded path before running computeSimulationAdjustment. This fires before any early-continue (no theater match, no candidate packet, zero confidence), so reloaded forecast-eval.json paths from applyPostSimulationRescore never retain stale simulation metadata from a prior cycle. P3: corrected simPathConfidence JSDoc in SimulationAdjustmentDetail and SimulationSignal — absent/non-finite → 1.0 fallback, explicit 0 preserved as 0 (was: "missing/null/zero fall back to 1.0"). Tests: T-O7 (zero-confidence match clears stale fields), T-O8 (no-theater match clears stale fields). 272 passing. 🤖 Generated with Claude Sonnet 4.6 via Claude Code (https://claude.ai/claude-code) + Compound Engineering v2.49.0 Co-Authored-By: Claude Sonnet 4.6 (200K context) <noreply@anthropic.com>
…itive adjustments
Summary
+0.08and+0.04simulation bonuses now scale bysimPathConfidencefromSimulationTopPath.confidence. Missing/null/zero confidence falls back to 1.0 (conservative — old artifacts not penalized). Negative adjustments (-0.12/-0.15) remain flat (structural, not sim-confidence-dependent).applySimulationMergeattaches a compactsimulationSignalobject (backed,adjustmentDelta,channelSource,demoted,simPathConfidence) to eachExpandedPathwhen adjustment ≠ 0. Written to R2 trace artifacts via existingwriteForecastTraceArtifacts.simPathConfidencefield toSimulationAdjustmentDetailso adjustments per path are fully inspectable in trace output.stateSummary.actors("Commodity traders") haveactorOverlapCount=0by design — only named geo-political actor theaters (maritime/security) fire the +0.04 bonus.Closes out the last two "Next (planned)" items from
docs/internal/wm-mirofish-gap.md.ForecastPanel UI chip deferred:
simulationSignallives in R2 trace artifacts. Surfacing it as a UI chip requires a proto field addition +buf generate+ prediction-to-path plumbing. Tracked for a follow-up PR.Testing
Post-Deploy Monitoring & Validation
simulation-outcome-*.json— checksimulationSignalfields on paths with non-zerosimulationAdjustmentbucketChannelMatch=truehavesimulationSignal.backed=true,adjustmentDeltamatchessimulationAdjustment,simPathConfidence< 1.0 when sim output includes fractional confidencesimulationSignalabsent on a path that has non-zerosimulationAdjustment(would indicateapplySimulationMergeearly-exit regression)🤖 Generated with Claude Sonnet 4.6 (200K context) via Claude Code