chore(ai): re-baseline the decision-cost perf gate onto post-#6826 cost - #7053
Conversation
#7044 refreshed the baseline onto the #6826 regression itself (layers_full_eval 3495 -> 15877). Once #7049 fixed that regression the gate carried ~3x slack -- real cost 5464 against a 15877 baseline under a ~6% band -- so it could no longer catch a new decision-cost regression. Re-measured on a clean worktree against an immutable card-data snapshot (scoped card_data_hash 670a4a14, covering 46/46 scenario deck cards): counter | old | new | threshold layers_full_eval | 15877 | 5464 | 5801 state_clone_for_legality | 19342 | 11099 | 11717 restriction_static_mode_gate_scans | 155747 | 110891 | 116499 sba_battlefield_snapshot_builds | 27462 | 19626 | 20671 scripts/validate-ai-perf-reproducibility.sh: PASSED (margin+band). 25/25 band runs clean, "0 OVER-MARGIN of 29 counters", and worst_current == baseline on every counter across 125 cold processes. CI budget: T_run_max 90s * 2.5 + T_build ~630s = ~14.3 min < 25 min. (The script reported T_build=1s from a warm cache; the 630s figure is the cold isolated server-release build measured separately, which is the conservative ceiling the check intends.) git_sha stamps 7f5f0b4, where the 25-run validation ran. Re-confirmed against f26f4e3 (current main, including #7051 and #7017): 0 FAIL, 29 PASS, +0 on every counter.
📝 WalkthroughWalkthroughThe performance baseline records a new Git SHA, updated scan and evaluation counters, and a lower wall-clock measurement. ChangesPerformance baseline refresh
Estimated code review effort: 1 (Trivial) | ~5 minutes Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@crates/phase-ai/baselines/perf-baseline.json`:
- Around line 19-42: Limit the baseline refresh in the performance baseline JSON
to the four counters named by the PR objective, along with the intended
threshold and baseline SHA updates. Restore unrelated values such as
crew_eligibility_scans, layers_escalated, layers_incremental,
legend_rule_mode_gate_scans, mana_aura_trigger_scans, and
sba_empty_battlefield_short_circuits; preserve the existing schema without
adding undocumented counter changes.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: ba0beb4f-33a5-4d5e-91ae-a09f8483d8bb
📒 Files selected for processing (1)
crates/phase-ai/baselines/perf-baseline.json
| "crew_eligibility_scans": 11015, | ||
| "granted_ability_provider_scans": 0, | ||
| "layers_escalated": 73, | ||
| "layers_full_eval": 15877, | ||
| "layers_full_eval": 5464, | ||
| "layers_incremental": 395, | ||
| "legal_actions_spell_cost_sweeps": 10, | ||
| "legend_rule_mode_gate_scans": 27535, | ||
| "mana_aura_trigger_scans": 53208, | ||
| "legend_rule_mode_gate_scans": 19703, | ||
| "mana_aura_trigger_scans": 40836, | ||
| "mana_display_sweeps": 247, | ||
| "mana_display_swept_objects": 2455, | ||
| "priority_cast_probe_builds": 10, | ||
| "restriction_static_exact_scans": 0, | ||
| "restriction_static_mode_gate_scans": 155747, | ||
| "sba_battlefield_snapshot_builds": 27462, | ||
| "sba_empty_battlefield_short_circuits": 35, | ||
| "restriction_static_mode_gate_scans": 110891, | ||
| "sba_battlefield_snapshot_builds": 19626, | ||
| "sba_empty_battlefield_short_circuits": 31, | ||
| "spell_keyword_grant_scans": 0, | ||
| "stack_batch_candidates": 0, | ||
| "stack_batch_observer_refusals": 0, | ||
| "stack_batch_plans": 0, | ||
| "stack_batched_entries": 0, | ||
| "stack_inert_noop_batches": 0, | ||
| "stack_inert_noop_entries": 0, | ||
| "state_clone_for_legality": 19342, | ||
| "state_clone_for_legality": 11099, | ||
| "static_full_scans": 0 |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | 🏗️ Heavy lift
Keep the baseline refresh limited to the intended counters.
The PR objective names four counters, but Lines [19-42] also change values such as crew_eligibility_scans, layers_escalated, layers_incremental, legend_rule_mode_gate_scans, mana_aura_trigger_scans, and sba_empty_battlefield_short_circuits. These unrelated changes can move regression gates without corresponding measurement evidence.
Restore unrelated counters, or document and validate each additional update with the same immutable snapshot and paired-seed process.
As per path instructions: preserve the existing schema and make only the intended measured-value, threshold, and baseline SHA updates.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@crates/phase-ai/baselines/perf-baseline.json` around lines 19 - 42, Limit the
baseline refresh in the performance baseline JSON to the four counters named by
the PR objective, along with the intended threshold and baseline SHA updates.
Restore unrelated values such as crew_eligibility_scans, layers_escalated,
layers_incremental, legend_rule_mode_gate_scans, mana_aura_trigger_scans, and
sba_empty_battlefield_short_circuits; preserve the existing schema without
adding undocumented counter changes.
Source: Path instructions
#7044 refreshed the baseline onto the #6826 regression itself
(layers_full_eval 3495 -> 15877). Once #7049 fixed that regression the
gate carried ~3x slack -- real cost 5464 against a 15877 baseline under a
~6% band -- so it could no longer catch a new decision-cost regression.
Re-measured on a clean worktree against an immutable card-data snapshot
(scoped card_data_hash 670a4a14, covering 46/46 scenario deck cards):
counter | old | new | threshold
layers_full_eval | 15877 | 5464 | 5801
state_clone_for_legality | 19342 | 11099 | 11717
restriction_static_mode_gate_scans | 155747 | 110891 | 116499
sba_battlefield_snapshot_builds | 27462 | 19626 | 20671
scripts/validate-ai-perf-reproducibility.sh: PASSED (margin+band).
25/25 band runs clean, "0 OVER-MARGIN of 29 counters", and
worst_current == baseline on every counter across 125 cold processes.
CI budget: T_run_max 90s * 2.5 + T_build ~630s = ~14.3 min < 25 min.
(The script reported T_build=1s from a warm cache; the 630s figure is the
cold isolated server-release build measured separately, which is the
conservative ceiling the check intends.)
git_sha stamps 7f5f0b4, where the 25-run validation ran. Re-confirmed
against f26f4e3 (current main, including #7051 and #7017):
0 FAIL, 29 PASS, +0 on every counter.
Summary by CodeRabbit