Skip to content

chore(ai): re-baseline the decision-cost perf gate onto post-#6826 cost - #7053

Merged
matthewevans merged 1 commit into
mainfrom
ship/perf-baseline-post-6826
Aug 6, 2026
Merged

chore(ai): re-baseline the decision-cost perf gate onto post-#6826 cost#7053
matthewevans merged 1 commit into
mainfrom
ship/perf-baseline-post-6826

Conversation

@matthewevans

@matthewevans matthewevans commented Aug 6, 2026

Copy link
Copy Markdown
Member

#7044 refreshed the baseline onto the #6826 regression itself
(layers_full_eval 3495 -> 15877). Once #7049 fixed that regression the
gate carried ~3x slack -- real cost 5464 against a 15877 baseline under a
~6% band -- so it could no longer catch a new decision-cost regression.

Re-measured on a clean worktree against an immutable card-data snapshot
(scoped card_data_hash 670a4a14, covering 46/46 scenario deck cards):

counter | old | new | threshold
layers_full_eval | 15877 | 5464 | 5801
state_clone_for_legality | 19342 | 11099 | 11717
restriction_static_mode_gate_scans | 155747 | 110891 | 116499
sba_battlefield_snapshot_builds | 27462 | 19626 | 20671

scripts/validate-ai-perf-reproducibility.sh: PASSED (margin+band).
25/25 band runs clean, "0 OVER-MARGIN of 29 counters", and
worst_current == baseline on every counter across 125 cold processes.

CI budget: T_run_max 90s * 2.5 + T_build ~630s = ~14.3 min < 25 min.
(The script reported T_build=1s from a warm cache; the 630s figure is the
cold isolated server-release build measured separately, which is the
conservative ceiling the check intends.)

git_sha stamps 7f5f0b4, where the 25-run validation ran. Re-confirmed
against f26f4e3 (current main, including #7051 and #7017):
0 FAIL, 29 PASS, +0 on every counter.

Summary by CodeRabbit

  • Chores
    • Updated performance benchmarks to reflect current application behavior.
    • Refined recorded evaluation and scanning metrics for more accurate performance comparisons.
    • Refreshed timing measurements used to track performance over time.

#7044 refreshed the baseline onto the #6826 regression itself
(layers_full_eval 3495 -> 15877). Once #7049 fixed that regression the
gate carried ~3x slack -- real cost 5464 against a 15877 baseline under a
~6% band -- so it could no longer catch a new decision-cost regression.

Re-measured on a clean worktree against an immutable card-data snapshot
(scoped card_data_hash 670a4a14, covering 46/46 scenario deck cards):

  counter                             | old    | new    | threshold
  layers_full_eval                    |  15877 |   5464 |      5801
  state_clone_for_legality            |  19342 |  11099 |     11717
  restriction_static_mode_gate_scans  | 155747 | 110891 |    116499
  sba_battlefield_snapshot_builds     |  27462 |  19626 |     20671

scripts/validate-ai-perf-reproducibility.sh: PASSED (margin+band).
25/25 band runs clean, "0 OVER-MARGIN of 29 counters", and
worst_current == baseline on every counter across 125 cold processes.

CI budget: T_run_max 90s * 2.5 + T_build ~630s = ~14.3 min < 25 min.
(The script reported T_build=1s from a warm cache; the 630s figure is the
cold isolated server-release build measured separately, which is the
conservative ceiling the check intends.)

git_sha stamps 7f5f0b4, where the 25-run validation ran. Re-confirmed
against f26f4e3 (current main, including #7051 and #7017):
0 FAIL, 29 PASS, +0 on every counter.
@matthewevans
matthewevans enabled auto-merge August 6, 2026 02:31
@coderabbitai

coderabbitai Bot commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

📝 Walkthrough

Walkthrough

The performance baseline records a new Git SHA, updated scan and evaluation counters, and a lower wall-clock measurement.

Changes

Performance baseline refresh

Layer / File(s) Summary
Refresh baseline measurements
crates/phase-ai/baselines/perf-baseline.json
Updates the recorded Git SHA, performance counters, and wall-clock baseline.

Estimated code review effort: 1 (Trivial) | ~5 minutes

Possibly related PRs

Suggested reviewers: mcbradd

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly and concisely describes the re-baselining of the AI decision-cost performance gate.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch ship/perf-baseline-post-6826

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@crates/phase-ai/baselines/perf-baseline.json`:
- Around line 19-42: Limit the baseline refresh in the performance baseline JSON
to the four counters named by the PR objective, along with the intended
threshold and baseline SHA updates. Restore unrelated values such as
crew_eligibility_scans, layers_escalated, layers_incremental,
legend_rule_mode_gate_scans, mana_aura_trigger_scans, and
sba_empty_battlefield_short_circuits; preserve the existing schema without
adding undocumented counter changes.
🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: ba0beb4f-33a5-4d5e-91ae-a09f8483d8bb

📥 Commits

Reviewing files that changed from the base of the PR and between f26f4e3 and 52fdce5.

📒 Files selected for processing (1)
  • crates/phase-ai/baselines/perf-baseline.json

Comment on lines +19 to 42
"crew_eligibility_scans": 11015,
"granted_ability_provider_scans": 0,
"layers_escalated": 73,
"layers_full_eval": 15877,
"layers_full_eval": 5464,
"layers_incremental": 395,
"legal_actions_spell_cost_sweeps": 10,
"legend_rule_mode_gate_scans": 27535,
"mana_aura_trigger_scans": 53208,
"legend_rule_mode_gate_scans": 19703,
"mana_aura_trigger_scans": 40836,
"mana_display_sweeps": 247,
"mana_display_swept_objects": 2455,
"priority_cast_probe_builds": 10,
"restriction_static_exact_scans": 0,
"restriction_static_mode_gate_scans": 155747,
"sba_battlefield_snapshot_builds": 27462,
"sba_empty_battlefield_short_circuits": 35,
"restriction_static_mode_gate_scans": 110891,
"sba_battlefield_snapshot_builds": 19626,
"sba_empty_battlefield_short_circuits": 31,
"spell_keyword_grant_scans": 0,
"stack_batch_candidates": 0,
"stack_batch_observer_refusals": 0,
"stack_batch_plans": 0,
"stack_batched_entries": 0,
"stack_inert_noop_batches": 0,
"stack_inert_noop_entries": 0,
"state_clone_for_legality": 19342,
"state_clone_for_legality": 11099,
"static_full_scans": 0

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🗄️ Data Integrity & Integration | 🟠 Major | 🏗️ Heavy lift

Keep the baseline refresh limited to the intended counters.

The PR objective names four counters, but Lines [19-42] also change values such as crew_eligibility_scans, layers_escalated, layers_incremental, legend_rule_mode_gate_scans, mana_aura_trigger_scans, and sba_empty_battlefield_short_circuits. These unrelated changes can move regression gates without corresponding measurement evidence.

Restore unrelated counters, or document and validate each additional update with the same immutable snapshot and paired-seed process.

As per path instructions: preserve the existing schema and make only the intended measured-value, threshold, and baseline SHA updates.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/phase-ai/baselines/perf-baseline.json` around lines 19 - 42, Limit the
baseline refresh in the performance baseline JSON to the four counters named by
the PR objective, along with the intended threshold and baseline SHA updates.
Restore unrelated values such as crew_eligibility_scans, layers_escalated,
layers_incremental, legend_rule_mode_gate_scans, mana_aura_trigger_scans, and
sba_empty_battlefield_short_circuits; preserve the existing schema without
adding undocumented counter changes.

Source: Path instructions

@matthewevans
matthewevans added this pull request to the merge queue Aug 6, 2026
Merged via the queue into main with commit 881c1c8 Aug 6, 2026
17 checks passed
@matthewevans
matthewevans deleted the ship/perf-baseline-post-6826 branch August 6, 2026 03:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant