Skip to content

Epic: independent changes land on main concurrently, with only overlapping work serialized #2934

Open
10 of 19 issues completed

Description

@schlessera

Selected outcome — conditional concurrency and reviewed recovery PRs

Several workers, each running several agents, can land independent changes on main concurrently. The proof obligations stay what they are today; what changes is where each one is enforced. Only work that actually conflicts with another candidate is serialized behind it: overlapping affected closures, or textual or structural conflicts between in-flight branches. A failure on main is detected and routed into the incident workflow without waiting for a human to notice. After a quick evidence-based assessment, the incident owner prepares and pushes either a focused fix PR or a revert PR; either follows normal review, required proof and merge procedures. The repository records the measured ceiling of the current model and the measured effect of every change made under this epic.

Motivation

The goal is throughput for the team as a whole, not for one worker. Most published agentic workflows assume a single developer running agents against a local checkout with fast loops. That works until several people each run several agents against one repository. At that point the bottleneck is no longer writing code. It is getting verified changes into main, the one place every worker's output has to pass through.

Adding agents only helps if three things hold:

  • independent changes do not wait for each other at integration;
  • proof is not repeated every time someone else merges;
  • collisions show up while work is in flight, not when it reaches main.

Each item in the breakdown removes one way the current model serializes work that does not actually conflict. It keeps the existing guarantees and changes where each one is enforced.

Why now

Merge volume motivates measuring the current coordination model. Available observations do not yet establish its capacity ceiling:

  • Observed volume. Merges into main per day: 9–15 on 2026-10-18 through 10-21, then 46, 39, 38 on 2026-10-22 through 10-24. That is 185 merges in the week since 2026-10-18 (read from gh pr list --state merged).

  • One candidate at a time, by policy. Every ordinary confirmation and merge is supposed to go through one repository-wide exact-candidate reservation, so only one candidate can be in confirmation at any moment (docs/decisions/ci-utility.md:1286-1308). A lease pins the exact base SHA and runs for up to 60 minutes, covering queue, proof and merge.

  • Every merge restarts the others. Base advancement requires a new confirmation request for every other candidate (.agents/skills/work-issue/SKILL.md:283-286). So each merge invalidates the proof of everyone waiting.

  • Illustrative serial bound; actual constraint unmeasured. A continuously occupied, strictly serial lease with a 25–30-minute confirmation-plus-merge service time would admit roughly 48–58 merges/day before failures and reconfirmation effects. Those inputs are assumptions, not measured capacity. Existing observations do not isolate caller wait, queue/proof fractions, base-advance reconfirmations or active-agent demand; task 1 must measure them. Runner price alone does not establish the dominant constraint.

  • Practice diverged from policy. The reservation ref records 45 acquisitions from 2026-10-19 to 2026-10-23 12:43 UTC+2 and none since, while about 100 squashes landed; observed occupancy was 4–6%. A recent red-main fix waited behind unrelated PRs for local proof resources until the maintainer had it merged with every check bypassed.

  • Work is concentrated and the proof system is in flux. Of 177 squashes since 2026-10-18, 88 touched packages/ui-server and 57 packages/ui-sdk, so in-flight closures rarely are disjoint. About a quarter of squashes since 2026-10-14 changed CI, test harness, sharding or timeouts. Long system-wide suites run concurrently on one host time out, and agents skip them to proceed. See the 2026-10-25 epic state comment.

This epic builds on #2476, which built the measurement collector and is closed, and on #2111, which introduced the reservation and is closed. It does not reopen either.

Repository measurements and failure attribution — 2026-10-25

These are repository observations, separate from the research survey and from a measured concurrency ceiling. Both delivery cohorts were recomputed with the same reporter at 6814f2ce5d3ea7f626284ab28b3c230428d06630, using normalized GitHub Actions attempts, PR lifecycle and exact squash identities. Overlapping archives are deduplicated. The fixed baseline is 2026-10-14 17:19:18 UTC through 2026-10-21 17:19:18 UTC; the subsequent cohort ends 2026-10-25 08:35:58 UTC. These are disjoint delivery cohorts, not a controlled policy experiment.

Measure Baseline Subsequent cohort
Merged PRs 271 128
Merges/day 38.7 35.2
First-pass green integrations/day 16.4 17.9
First-main green / known outcomes 115/271 (42.4%) 65/120 (54.2%); 8 unknown
Same-squash observed-green integrations 115 68
Observed runner-minutes / observed-green integration 742.5 359.0
PR creation → same-squash green main, median / p90 107 min / 23.8 h (n=115) 84 min / 12.9 h (n=68)
Recorded ready → merge, median / p90 16.9 / 34.6 min (n=161) 3 / 197 seconds (n=123)
Distinct PR verification overlap, mean while active / peak 4.1 / 31 1.2 / 8
Observed authorized-reservation occupancy of interval 5.52% 4.22%

Runner consumption includes observed failed/cancelled work for the delivery cohort; it is neither billed cost nor necessarily complete lifetime cost. The observed cost per green integration is about 52% lower, while green throughput is about 9% higher. This does not establish causality, sustainable capacity, or equivalent workload. PR-verification overlap is not active-agent concurrency. Ready timestamps do not split lease wait from proof; a late ready transition can exclude earlier waiting. Reservation occupancy comes from protocol/history observations across changing policy epochs, not a saturated demand experiment or a measurement of every caller's wait.

The subsequent cohort's first-main outcomes were 65 success, 49 failure, 6 cancellation, 8 unknown. A review of the 49 failed attempts and 83 failed primary check jobs attributes all failed checks in 26 attempts, some checks in 9, and leaves 14 unresolved. Aggregate failures and intentional negative controls were excluded as additional causes. Confirmed product mechanisms affect 15 attempts, test/fixture mechanisms 29, and setup infrastructure 1; these sets overlap and must not be summed. Sixty job-specific, evidence-linked failure-classified annotations were recorded for ongoing reports.

Largest recurring mechanisms:

These counts overlap. The issues document repairs for those historical mechanisms; this measurement pass did not execute new reproductions or claim every current variant fixed. At the October 25 review, open attribution included three native automatic-index aborts (#2921), raster parity (#2469), timer ownership after dirty mount disposal (#2894), tracker completion (#2900), and owned-loop cleanup (#2420). Similar error strings alone do not establish shared causes. The October 26 update below records the later product attribution and repair of #2900.

Implications for this epic's scoped tasks

  • Current Phase B task 9 (baseline): the serial 48–58/day example remains an assumed bound, not measured capacity. Add explicit reservation-request/admission/refusal, worker work/wait and reconfirmation-reason events before attributing the ceiling to lease serialization. Preserve workload, proof-policy and runner cohorts.
  • Current Phase A task 7 (ci: record a red main centrally, stop doomed proof and file one incident issue #2942) and Phase B task 10 (culprit/revert): a failed main check alone is insufficient evidence that the latest squash introduced a product regression. Distinguish inherited red main, product regressions, fixture defects, setup failures and unknowns before automated culprit/revert decisions. Repeated instances of one known failure are not independent bad PRs. This is evidence for scoping; no new policy ruling is implied.
  • Current Phase B task 12 (re-measure): compare first-pass green deliveries, tail latency, consumed runner time, recovery/rework and incidents together. Incident rate and actual active-agent capacity remain unknown without their event sources. The current classifiedFailedAttempts summary means at least one annotated job in an attempt, not complete causal attribution.
  • Pre-merge escape attribution remains open: this review identifies failure mechanisms, but does not yet establish which affected-selection, candidate-input or proof-enforcement boundary missed each one. Nor does it independently audit all formal proof guarantees. Recent squash-to-receipt comparisons were 4 matches / 0 differences / 124 unknown; absent resource logs are an evidence gap, not proof of a mismatch.

Source provenance, representative job links, every family count and the 49-run classification ledger are in the dated evidence supplement. No proof, branch, reservation or merge policy was changed by this measurement work.

Rolling observation and attribution update — 2026-10-26

A separate rolling delivery cohort, 2026-10-23 07:34:28 UTC–2026-10-26 07:46:10 UTC, contains 103 merged PRs: 53 first-main green, 43 failed, 7 unknown; 53/96 = 55.2% green among known outcomes and 17.62 first-pass green integrations/day. Available runner consumption is 299.78 runner-minutes per observed-green integration (58 same-squash green integrations); create→green median/p90 167.2/682.5 minutes, merge→green 18.1/21.6 minutes, n=58. This overlaps the earlier subsequent cohort and uses reporter 71503546feebd725040b315de9dd34ad97027e1d; it is not an independent daily sample or a controlled savings comparison.

Six additional failed first-main attempts were reviewed: three test faults (two handoff-frame assertions, #2935; one wrong-phase cancellation control, #2958), one product recovery/Activity-commit ordering defect (#2900/#2983), and two unresolved causes (WebKit menu absence, #2946; runner shutdown/exit137, #2945). The tracker correction landed after the failing squash; it is not evidence of a fixture fault. Within this rolling cohort's 43 failed attempts, attribution is 27 full / 5 partial / 11 unresolved. Across the original 49 plus six new attempts it is 30 full / 9 partial / 16 unresolved, with 64 job-level annotations. Cause counts overlap; run association does not establish the introducing commit.

The latest hourly artifact covers 631 attempts with no missing/incomplete attempts or acquisition errors; 23 additional successful artifacts were preserved. Joined historical data retains four incomplete attempts from older partial acquisitions. Scheduled resource-log omission leaves all 103 candidate-receipt/squash comparisons unknown; this is a measurement gap, not proof mismatch or proof of skipped confirmation. Actual workers, incident rates, caller wait and saturated capacity remain unknown. The rolling interval's 1.38% observed authorized-reservation occupancy cannot establish the lease bottleneck.

This observation does not qualify the Phase B gate: it is a mixed-policy rolling interval of about three days, below 80% known-outcome first-main green, with attribution incomplete. Seven consecutive days in one frozen epoch and complete incident attribution remain required. #2937's escape audit should include the six new attempts; the pre-merge selection/confirmation/bypass boundary is not settled by failure classification. The October 26 evidence supplement records sources, the six-job ledger and limitations.

Weekly review and rolling update — 2026-10-27

The frozen 2026-10-19 07:46 UTC–2026-10-26 07:46 UTC review changes the balance of evidence: versus the fixed October 14–21 baseline, observed runner-minutes per green integration are 742.48 → 372.48 (−49.8%), and known-outcome first-main green 115/271 (42.4%) → 100/170 (58.8%, +16.4 pp; 11 unknown). However, first-pass green integrations/day are 16.43 → 14.29 (−13.0%), create→same-squash-green median 106.95 → 145.20 min (+35.8%, n=115/105) and p90 23.76 → 29.28 h (+23.2%). Thus observed runner efficiency improved; increased reliable parallel throughput is not established. Windows overlap, workloads/policies differ, and cost is available runner time rather than billing, CPU or complete lifetime cost. The preceding October 12–19 week lacks approximately 2.4 days of CI history; no complete week-on-week effect can be inferred.

The later October 24 07:46 UTC–October 27 07:40 UTC rolling cohort is separate: 87 merged PRs, first-main 51 green / 35 failed / 1 unknown (59.3% among 86 known), 17.02 green integrations/day and 296.08 runner-minutes per observed-green integration. Do not treat overlapping rolling cohorts as independent daily samples. The latest artifact has 338 attempts, no missing/incomplete snapshots or collection errors; 23 additional artifacts preserve 19 completed and four earlier partial manifests. Four incomplete snapshots from an older partial acquisition remain visible in joined history.

Five additional failures were reviewed: three missing-zsh test-prerequisite faults (#3025), one unresolved SIGKILL descendant observation (#3027), and one unresolved six-entry timer-observer residue (#3020). The optional-zsh availability gate means absent-zsh hosts do not execute those assertions. The observer correction #3031 does not establish the natural six-handle cause. The current rolling failed cohort has 21 full / 4 partial / 10 unresolved attributions (35 attempts); cumulative review is 33 full / 9 partial / 18 unresolved (60 attempts, 94 primary jobs, 67 job annotations). #2937's pre-merge escape audit should include the additional five; diagnosis is distinct from selection, candidate, confirmation or bypass attribution.

The weekly audit sampled 20 older main runs without finding new attempt numbers, not exhaustive older-rerun proof. Weekly receipt/squash comparisons are 26 matches / 0 differences / 155 unknown; no independent guarantee is claimed. Work/wait/incident instrumentation and a frozen proof-policy epoch remain missing. These observations do not qualify the existing Phase B gate or establish a saturated lease bottleneck. The October 27 supplement contains exact endpoints, denominator distinctions, source provenance, the five-job ledger and the modeled 2.6-full-run scenario.

Acquisition completeness and rolling observation — 2026-10-28

The newest daily evidence exposes a current measurement limit: 24 successful workflow artifacts contain 13 completed / 11 partial collections. Latest partial observations extend to October 28 07:33 UTC, but the last fully validated archive ends at 04:33 UTC (300 attempts, no missing/incomplete snapshots). A 30-request resources/reservations backfill also stopped partial; its output is retained and the complete-coverage cursor was not advanced. A green measurement workflow is not evidence of complete acquisition.

Provisional rolling interval October 25 07:40 UTC–October 28 04:33 UTC: 53 observed merged PRs, 39/53 first-main green (73.6%), 13.59 first-pass greens/day, and 311.02 observed runner-minutes per observed-green integration (43 same-squash greens). Joined evidence retains seven lifecycle acquisition errors, 50 resource problems, four older incomplete snapshots and two unattributed CI attempts. These all-input errors and partial recent acquisition can limit cohort/cost completeness; the changed rolling percentage is not a controlled improvement, a full lifetime-cost measurement or a qualified frozen seven-day epoch.

Two new failures remain unresolved: #3041 fails while observing an explicit invocation's child before its cancellation/reap assertions; #3073 refuses before payload start on invalid worker mount record (#2701), without the failed mountinfo/process-state evidence needed for attribution. Current rolling failed-attempt attribution is 7 full / 0 partial / 7 unresolved (14); cumulative reviewed history 33 full / 9 partial / 20 unresolved (62 attempts, 96 primary jobs). The existing 67 job annotations remain unchanged. #2937's escape audit should include these two failures without inferring the introducing commit or a cause from the test name. Existing rulings and gates remain unchanged; the linked supplement supplies exact endpoints, public job links, code/schema provenance and coverage limits.

Acquisition lag and bounded failure review — 2026-10-29

The October 29 evidence update preserves 24 new successful workflow artifacts: 1 complete / 23 partial collection manifests. Fully validated acquisition now ends at October 28 13:33 UTC (321 attempts), while the newest partial archive has evidence through October 29 07:33 UTC (322 attempts). The completeness lag has widened to approximately 18 hours, from about three hours in the prior update. A 30-request resources/reservations backfill remained partial; no complete cursor advanced from it. Successful workflow status is not complete measurement coverage.

Provisional rolling analysis October 26 04:33 UTC–October 28 13:33 UTC: 39 observed merges, 29/39 first-main green (74.4%), 12.21 first-pass greens/day, and 368.89 observed runner-minutes per observed-green integration (30 same-squash greens). This excludes the freshest approximately 18-hour partial interval. Joined evidence has nine lifecycle acquisition errors, 85 resource problems, four older incomplete snapshots and three unattributed CI attempts; counts/costs can be incomplete. These overlapping, mixed-policy observations do not establish a causal improvement, complete lifetime cost, a saturated bottleneck or a qualified frozen epoch.

Three new primary failures remain unresolved: #3082 does not observe a complete journal frame after releasing a held turn; #3083 reaches both queued states but not answered after reconnection (distinct from #3034's earlier queued request-ID mismatch); #3087 fails exact light raster parity within #2469's scope, with no decoded new captures or causal diagnosis. The last occurs after the primary analysis cutoff and is only in fresher/cumulative review. Primary failed-attempt attribution is 4 full / 0 partial / 6 unresolved (10); cumulative review 33 full / 9 partial / 23 unresolved (65 attempts, 99 primary jobs). All 67 previous annotations remain unchanged. Preserve the introducing-commit and pre-merge escape questions separately and retain the existing maintainer deferral and proof/merge rulings.

Recent acquisition recovery — 2026-10-30

The October 30 evidence closes the recent completeness gap: 24 new artifacts contain 9 complete / 15 partial manifests. The latest validated archive has 425 attempts, no missing/incomplete snapshots and completed lifecycle/reservations, extending the cursor to October 30 07:33 UTC. Its overlap covers the prior gap; no duplicate manual backfill was required. Historical partial scopes and seven joined lifecycle errors, 85 resource problems, four older incomplete snapshots and three unattributed attempts remain visible.

Rolling interval October 26 13:33 UTC–October 30 07:33 UTC: 51 observed merges, first-main 37 green / 13 failed / 1 cancelled (72.55% among 51 known outcomes), 9.87 first-pass greens/day, and 449.37 observed runner-minutes per observed-green integration (39 same-squash greens). Window length, workload and policy differ from prior rolling samples; no causal savings, saturated capacity or frozen-epoch qualification follows. Three new primary failures include one source-backed production inspector error-normalization defect (#3174/#3175: raw EPIPE instead of unavailable), plus unresolved continuity scene preparation and hygiene focus observations. Failed-attempt attribution is 4 full / 0 partial / 9 unresolved (13); cumulative review 34 full / 9 partial / 25 unresolved (68 attempts, 102 primary jobs), with 68 append-only job annotations. Existing deferral, proof obligations and merge rulings remain unchanged.

Current rolling observation and failure attribution — 2026-10-31

The October 31 evidence supplement records complete acquisition through 2026-10-31 07:33:09.195 UTC: 24 additional artifacts, 11 completed / 13 partial manifests, including partial artifacts from two failed collection workflows. The newest validated archive contains 563 attempts with no missing/incomplete snapshots. Earlier joined gaps remain explicit; telemetry recovery does not erase historical errors.

The overlapping rolling delivery cohort 2026-10-28 07:33:36.221 UTC–2026-10-31 07:33:09.195 UTC has 41 PRs: 32 first-main green / 8 failed / 1 cancelled / 0 unknown (78.0%), 10.67 first-pass green/day and 530.56 observed runner-minutes per observed-green integration (33 same-squash green). Create→green median/p90 are 111.45/813.50 minutes (n=33). These changing rolling cohorts are not independent daily samples or a frozen policy epoch; this does not qualify the existing gate or establish causal optimization gains.

Two new primary-job reviews distinguish one confirmed fixture observation race (#3186, delivered #3193) from one unresolved native automatic-index abort plus cleanup EBUSY occurrence. Matching outer signatures do not establish shared causes with #2921/#2420. Rolling failed-attempt attribution is 2 full / 0 partial / 6 unresolved / 0 unreviewed (8); cumulative reviewed history is 35 full / 9 partial / 26 unresolved (70 failed attempts, 104 primary jobs, 69 append-only annotations). Historical failures remain failed. #2937's introducing-commit and pre-merge escape boundary remains a separate audit. Actual worker activity, caller wait, incidents and independent guarantee satisfaction remain unknown from these statistics. Earlier dated observations, task hierarchy and the existing deferral/rulings remain unchanged.

Cutoff-limited rolling observation — 2026-11-01

The November 1 evidence supplement records 24 new artifacts: 13 complete / 11 partial manifests. Validated coverage ends 2026-10-31 23:33:28.071 UTC; newer partial evidence reaches 2026-11-01 07:33:15.281 UTC with lifecycle budget errors. A 30-request resource/reservation backfill remained partial and did not advance that cursor. Earlier gaps remain recorded.

The overlapping rolling delivery cohort 2026-10-29 07:33:09.195 UTC–2026-10-31 23:33:28.071 UTC has 32/39 first-main green (82.1%), seven failures, no cancelled/unknown outcomes, 12.00 first-pass green/day, 526.24 observed runner-minutes/green, create→green median/p90 119.53/1,053.83 minutes (n=32). Partial enrichment changes available cost attribution; this is neither a causal saving nor independent daily evidence. Above 80% in about 2.67 days does not qualify the frozen seven-day policy gate.

Two new failed primary jobs remain unresolved: schedule folder admission's invalid mount record (#2701 diagnostic context) and natural browser handoff timing of 748.5ms against <700ms (#1946 diagnostic context). No shared cause or fixture/product ownership is inferred. Rolling failed attribution is 2 full / 0 partial / 5 unresolved / 0 unreviewed; cumulative history is 35 full / 9 partial / 28 unresolved across 72 attempts and 106 primary jobs, with 69 job annotations. Introducing-commit/escape attribution remains separate #2937 work. Existing deferral, proof obligations and merge policy remain binding; actual agent activity, caller wait and incidents are unmeasured.

Acquisition lag and newer failure evidence — 2026-11-02

The November 2 supplement records 24 new successful workflow artifacts, all partial collections. Validated coverage remains 2026-10-31 23:33:28.071 UTC; newer partial observations reach 2026-11-02 07:33:03.599 UTC, about 32 hours later, with lifecycle budget errors and 14 observed jobs missing duration. A 30-request resource/reservation backfill remained partial despite exit 0 and did not advance the cursor. The current report's delivery interval ends at the stale cutoff; it cannot establish current reliable throughput or Phase B gate qualification. Earlier dated rolling figures remain historical observations.

Two newer browser failures are outside that primary analysis interval: missing native snap-back Animation (#3226) and missing artifact-error heading (#3260). #2444/#3227 delivered a controlled native observation correction for the snap-back failure class, while explicitly retaining the original run's unobserved timing cause. Both new natural occurrences remain unresolved, with no new cause events. Cumulative review: 74 failed first-main attempts / 108 primary jobs, 35 full / 9 partial / 30 unresolved, all 69 prior canonical annotations included; no known log backlog. The historical cutoff-limited cohort's five failed attempts are 2 full / 0 partial / 3 unresolved / 0 unreviewed. Failure-class evidence is separate from #2937's introducing-commit and pre-merge escape audit. Existing publication-start-gate reconciliation, task hierarchy, readiness, Phase B gate and merge policy are preserved.

Recovered acquisition and expanded evidence — 2026-11-03

The November 3 supplement records 24 new artifacts (2 complete / 22 partial). Validated creation-window coverage recovered through 2026-11-02 22:33:36.688 UTC, closing the older gap; about nine newer hours remain partial. Fourteen observed jobs still lack duration. The overlapping October 29 23:33:28.071–November 2 22:33:36.688 UTC cohort has 51/60 first-main green (85%), nine failed, no unknown/cancelled outcomes; failed attribution 2 full / 0 partial / 7 unresolved / 0 unreviewed. This mixed-policy rolling observation does not qualify the frozen seven-day gate.

Five new primary-job reviews across four failed attempts add one source-bound recording-test expectation diagnosis and one partial resource-precondition attribution. Other natural loading, file-switch and focus-frame mechanisms remain unresolved despite related fixture corrections. Cumulative 78 failed attempts / 113 primary jobs: 36 full / 10 partial / 32 unresolved, with 71 canonical annotations; two newer attempts are after the primary cutoff. The separately published #2937 audit concerns the original 49, not automatic escape attribution for all 78.

The frozen October 24–31 versus October 17–24 review reports first-main green 87/116 (75%, one unknown) versus 85/146 (58.2%, ten unknown). Green deliveries/day rise 2.4%, observed runner-minutes/green fall 6.2%, and p90 delivery latency falls; versus the fixed baseline, green deliveries/day remain 24.3% lower despite 46.6% lower observed runner-minutes/green. Workload/policy/runtime and coverage confound causality. Actual agent activity, caller wait, complete incident rates and independently validated guarantees remain unmeasured. Conditional concurrency, assessed fix/revert through normal review, existing gates and hierarchy remain unchanged by measurement.

Research findings

This is a survey of practitioner reports, papers and vendor data published up to 2026-10. Vendor-measured numbers are marked. Most published throughput figures count PRs, not outcomes.

Merge to main

  • A global lock does not survive agent scale. Anthropic reports about 2,500 merged changes a day in May 2026 and more than 30,000 a day by September. In their words, "a proper merge queue doesn't make much sense at agent scale"; they could not replicate a global repository lock. They moved verification to post-merge and pre-deploy testing, with automated bisect and revert, and agents resolve conflicts. No merger agent is described; one agent only monitors the merge queue's health. Human reviewers examine intent rather than every line. Source: https://www.anthropic.com/engineering/redesigning-source-control (2026-10-13). This motivates steps 2–5.

  • Big monorepos made the same trade long ago. Google's TAP runs a scoped presubmit, then batched postsubmit with automated culprit finding and auto-rollback, because running everything presubmit "is not feasible in latency and compute". Sources: https://research.google/pubs/what-breaks-google/ and the transition-prediction paper (ICST 2025), https://research.google/pubs/speculative-testing-at-google-with-transition-prediction/. This motivates step 5.

  • Serial queues have a hard ceiling. A serial queue with 30-minute CI allows about 48 merges a day, and only if nothing fails. Vendor data from Mergify (more than 200k merges across 477 orgs, 2026):

    • the rate of breaking main rises from 0.77% at 2–5 developers to 12.5% at 40+;
    • p99 queue wait is about 20 hours;
    • 94% of merges are still tested one at a time.

    Source: https://mergify.com/reports/state-of-merge-queues-2026.

  • Conflict-aware speculation and partitioning work. Uber's SubmitQueue builds likely orderings in parallel and prunes them with build-graph conflict analysis: CI resources −53%, CPU −44%, P95 wait −37%. 40–65% of its builds had been aborted early and wasted, and small changes got stuck behind large conflicting ones. Source: https://arxiv.org/abs/2501.03440 (ICSE 2025). Monorepo queues split into lanes by affected target, so disjoint changes never queue behind each other (Trunk parallel queues, Aviator affected-targets; vendor docs). Both depend on a trustworthy build or affected graph, which this repository already computes. This motivates steps 3–4.

  • CI concurrency is the limit, not compute. Harness, dogfooding its own platform (vendor), saw PRs and CI runs grow about 7x while contributors grew about 1.3x. Pipelines queued with compute free, so the limit was CI concurrency. Source: https://www.harness.io/blog/when-code-stops-being-the-bottleneck (2026-10-22).

  • More PRs, slower merges. A 2026 working paper on data from more than 700 firms reports PRs +23% after agent adoption but PR-to-merge time +49%. This figure comes from secondary coverage.

Collisions and coordination

  • In-flight collisions are the norm. Across 33,596 agent-authored PRs in 2,807 repositories, 95.0% belonged to co-active pairs in the paper’s one-week collaboration window; exact temporal overlap accounted for 79.4% of agent PRs. The window figure is not a claim of simultaneous open overlap. Textual conflict rates were 19.8% between PRs from the same agent and 41.7% between different agents, and 84% of conflicted files were source files. Build and semantic conflicts were not counted, so the true rate is higher. Source: https://arxiv.org/abs/2607.04697 (2026-07). AgenticFlict found a 27.67% conflict rate across more than 142k agent PRs (arXiv:2604.03551). This motivates step 6.
  • Agents coordinating as peers fails. Cursor had 20 agents claim work through shared locks; they reached the throughput of 2–3. Agents held locks too long or crashed while holding them, and optimistic concurrency made them risk-averse. A hierarchy worked: planners create tasks, workers only execute, a judge decides each cycle. A dedicated integrator role became the new bottleneck. Source: https://cursor.com/blog/scaling-agents (2026-01-14).
  • Messaging between agents doesn't fix correctness. In CooperBench, the evaluated agents average about 30% lower task success when working together than when one agent performs both tasks. This is a benchmark result, not a repository delivery/incident-rate measurement. Messaging reduced conflicts but not failures. Source: https://arxiv.org/abs/2601.13295 (2026-01).
  • High agent-PR volumes come from task shape. At organizations reporting very high volumes (more than 1,000 agent PRs a week), the work is mostly fleet migrations with one change per repository, or small one-shot fixes, so it is disjoint by construction. There is no published account of several people, each running several agents, coordinating within one area of one repository with anything beyond tracker assignment and PR review. That is the case this repository is in.
  • This is why the epic chooses detection and partitioning. Ordering is deterministic, and collisions are detected early rather than negotiated between agents.

CI and verification

  • Verification moves into the agent's own environment. Organizations running agents at volume have the agent run the deterministic checks in a pre-warmed full-stack environment before a PR exists, and cap CI round trips (one reports a maximum of 2). CI then confirms rather than iterates. This repository already does this with check:pr and ready-candidate confirmation.
  • Testing only what a change affects pays off. Meta's predictive test selection runs about a third of transitively dependent tests and still catches more than 99.9% of regressions. Source: https://engineering.fb.com/developer-tools/predictive-test-selection/. This repository's affected selection is the same lever, and the closure it computes is what steps 3–4 key on.
  • Flake exposure grows with volume. More integrations multiply exposure to flaky tests, not the flake rate. Uber found about 1k flaky tests among 600k in Go; one flake can invalidate several queued candidates. Source: https://www.uber.com/blog/flaky-tests-overhaul. This epic keeps the "fix, never rerun" rule but treats flake rate as a precondition to measure in step 1.
  • AI amplifies the system it runs in. DORA 2025, with about 5,000 respondents: AI adoption now goes with higher throughput and higher instability. Only teams with strong platforms, small batches and strong version-control practice gain throughput without the instability. Steps 5 and 8 are what keep the instability side visible.

What the research does not support

  • Agent-to-agent swarms or lock files at team scale: no adoption reports, and the lock-contention evidence above.
  • Vendor throughput claims for merge queues and stacking tools: not independently measured.
  • AI merge-conflict resolution and conflict prediction at team scale: previews and small tools only, with no outcome data. Step 6 is therefore detection, not automatic resolution.
  • PR count as a productivity measure: it rewards slicing, and merges get slower as PR counts rise. Step 8 compares green deliveries, tail latency, recovery work and incidents together, as docs/process/ci-throughput.md already requires.

Shared context

  • Reservation. scripts/integrate.ts (175 lines) holds the authoritative state in reservation.json on refs/heads/brain-integration-reservation. Each update uses exact --force-with-lease. There is no FIFO queue: a busy caller gets three bounded attempts (docs/decisions/ci-utility.md:1298-1319). The merge path re-reads the live candidate and verifies the squash parent and tree (docs/decisions/ci-utility.md:1321-1331).
  • No native merge queue. The repository is user-owned, so GitHub's merge queue is unavailable, and the policy forbids inventing one or requiring an ownership migration (docs/decisions/ci-utility.md:1274-1275).
  • Affected closures already exist. The repository already computes affected proof per change: import-graph unit selection, reverse manifest dependants, and declared browser proof inputs (docs/decisions/ci-utility.md:12-41, :83-100). These closures are what the partitioning below would key on.
  • Main is already proven after every merge. Main pushes keep complete affected proof, and scheduled exhaustive runs are an independent detector of omitted inputs (docs/decisions/ci-utility.md:36-38). The ci: measure reliable parallel integration throughput and proof costs #2476 closure records main-push CI as the hosted backstop.
  • Measurement and incident events exist. scripts/ci-timings.ts already reports first-pass green integrations per day, reservation occupancy, same-commit green, and failure-classified, incident and revert events (docs/process/ci-throughput.md:48-109).
  • Claims. Workers claim work by GitHub assignment plus a claim comment, after checking open PRs. Assignment is explicitly not an atomic lock (.agents/skills/work-issue/SKILL.md:80-96). Nothing compares the files or closures of in-flight branches against one another.
  • Orchestration. Several people each run an orchestrator agent managing several workers. Command is a tree (maintainer, orchestrators, workers); a worker answers to one orchestrator, and an orchestrator acts only within its own subtree. Host slots, the reservation and any red-main record are ledgers arbitrated by deterministic commands with durable state, not authorities; restrictions only add up, and cross-subtree contention is settled by ledger priority or the maintainer's revoke. Preemption cancels with cleanup and records preempted, which is neither a pass nor a failure. A worker without an orchestrator degrades to plain admission limits.
  • Integrator. The integrator is scripts/integrate.ts grown into a service (lanes, priority, admission, red-main record, proof checks, squash, revert), not an agent. Workers own their PRs end to end, including conflicts and fixing what they break.
  • Stacks. Native stacked candidates are not yet supported by the reservation command (docs/decisions/ci-utility.md:1333-1339).

Constraints and non-goals

  • No test, assertion, offline boundary, pinned image or proof category is removed or weakened. Statistics never authorize deleting a test or weakening a gate (docs/process/ci-throughput.md:131-133).
  • No branch protection, permission or ownership change. No native merge queue, and no external merge-queue service.
  • GitHub Actions remains the only CI provider (AGENTS.md:168-170).
  • "Fix flaky behavior; never rerun a real failure" stays (docs/decisions/ci-utility.md:1271-1272). This epic does not introduce flaky-test quarantine.
  • Any change to the exact-base confirmation rule or the reservation decision needs a recorded maintainer ruling before code. Tasks that depend on one carry needs: decision until it is given.
  • Deliberately out:
    • agent-to-agent messaging or a shared lock file between agents;
    • a dedicated integrator agent in the merge path: Cursor removed theirs because it created more bottlenecks than it solved, and it would blur who owns a failure;
    • long-lived epic integration branches;
    • batching several candidates into one confirmation, which needs bisection and multiplies the cost of a failure.

Maintainer rulings — conditional concurrency and assessed recovery PRs

Decision (maintainer, 2026-11-02): selected A — conditionally approve future concurrency. Independently qualified candidates may use concurrent confirmation and disjoint-base tolerance with reservations that serialize conflicting closures, only after all prerequisites below are met and recorded. This supersedes the October 25/29 deferral for the concurrency direction. It does not declare the gate met or activate a new integration path.

Conditions before concurrency rollout

  1. Preserve the existing stability gate: seven consecutive days in one proof-policy epoch, first-main green at or above 80%, and every red incident attributed. Record the frozen interval, numerator/denominator, complete acquisition/incident coverage and epoch. A short or stale rolling percentage, successful collector workflow, partial coverage or unresolved incident is not qualification. The current November 2 evidence has not established this gate.
  2. Task 9's measured baseline must show material lease waiting/reconfirmation cost that justifies the change, retaining actual caller wait, proof-policy/workload/runner cohorts and source completeness. If benefit is not established, retain the current path and record that result; approval is not a requirement to ship concurrency regardless of evidence.
  3. Qualify a complete affected-closure and global-input model before treating work as independent. Textually disjoint files alone are insufficient. Include transitive and shared proof/planner/configuration inputs; demonstrate conservative behavior for unknown, incomplete or changed graphs and global/broad changes. Unproven independence uses serial exact-base confirmation rather than optimistic admission. Overlapping closures or textual/structural conflicts are serialized.
  4. Scope and qualify the deterministic disjoint-base comparison and conflicting-closure reservations, including atomic competing acquisitions, ownership/expiry/revocation, cancellation, stale head/base and graph-generation races. A changed candidate head always requires its own proof. Record exact candidate heads, tested bases, live bases, intervening changes, closure/global-input identities, reservations and actual squash parents/trees so every accepted base advance is auditable.
  5. Keep all selected proof obligations, actual combined-tree main proof and incident ownership. Missing/cancelled/skipped selected jobs are not green, and a faster merge with unfinished main proof is not successful delivery. Preserve applicable red-main stop/refusal and recovery controls; this ruling provides no implicit route around a red main, no proof waiver and no permission/protection/credential expansion.
  6. Before implementation, fold the concrete qualification/proof/refusal cases into native child briefs and amend docs/decisions/ci-utility.md plus the relevant workflow/skill documentation. Classify any new deterministic command's supported --json/contract surface explicitly. Scope Phase B tasks from their prerequisite evidence rather than treating this epic as a coding task.

Until the gate, benefit, complete independence model and implementation qualification are recorded and delivered, today's exact-base confirmation and global reservation remain binding. Existing authorized stabilization, read-only measurement/collision detection and scoped design may continue within their current bounds. No runtime rollout, release commitment or existing proof weakening occurs in this decision review.

Recovery ruling — maintainer, 2026-11-02

Selected the maintainer's refinement: do a quick assessment, choose a fix PR or a revert PR, and use normal review either way. Recovery is not revert-only, and the incident controller does not directly auto-merge a recovery change.

  • The deterministic controller detects/records the red-main incident and routes it to the existing ownership workflow. The owning worker makes a short, bounded assessment using failed-job evidence, exact squash/base identities, prior main state and an appropriate focused reproduction. A red check or latest-squash position alone is not causal attribution.
  • Prefer a focused fix PR when the cause is sufficiently understood and a small correction can be prepared and proved promptly. Prefer a targeted revert PR when undoing an attributed change is the safer, quicker bounded recovery and the fix is uncertain or larger. Record the evidence, choice and tradeoff on the incident/PR. Do not prolong diagnosis indefinitely instead of selecting a supportable recovery.
  • Assess the current tree, dependent/later work, conflicts and side effects before proposing the change. Do not automatically roll back a range of later good squashes or erase unrelated work. If neither fix nor revert has a safe evidence-backed basis, preserve the red-main/unknown state and seek the necessary incident review rather than fabricate a culprit or force a speculative merge.
  • Push the concrete fix/revert PR through normal review, required proof and the recorded merge path, including the already-approved declared-red-main-fix procedure where applicable. No proof/check/review bypass or direct automatic recovery merge is introduced. The worker owns the PR end to end; the integrator remains deterministic, not a new integrator agent.
  • Keep one incident/owner, preserve duplicate/preemption/cancellation rules and record attempted/chosen recovery with existing incident evidence. Record actual revert events only for actual reverts. Do not mark recovered until the exact resulting main tree has the required green proof; a newly introduced failure is a new incident, not an unbounded fix/revert loop. Preserve reviewable owner/dependency impact in the incident/PR.

Before recovery implementation, update docs/decisions/ci-utility.md and the relevant incident/worker briefs with the assessment/choice, attribution, proof, conflict, uncertain-outcome and recovery/loop-stop cases. These are concrete design/proof tasks within the selected policy, not another unresolved choice between fix and revert.

Decision-review labels and scope

All product/policy choices in this review are recorded. Remove needs: decision; with no other needs: label on this issue, add agent-ready as requested for this review. This remains an epic: implementation is performed on scoped native children, and the epic remains excluded from coding pickup. The label does not establish the Phase B gate, qualify independence or authorize rollout. Phase A rulings remain settled on #2934/#2938/#2939; child membership, priorities and release commitments are unchanged.

Research evidence limit: the linked external workload reports and benchmarks do not establish semantic independence or a particular speedup in this repository. Earlier measurements and limitations remain preserved below/above; qualification and baseline evidence remain required.

Breakdown

Each item becomes a scoped task attached here; none is agent-ready until scoped. The reasoning for this order is in the 2026-10-25 epic state comment.

Phase A — stabilize proof and restore one merge path

The 0.44.0 publication start gate from the maintainer’s October 25 scheduling ruling is satisfied. The November 1 publication receipt records all nineteen packages from main a84d9bfbcb4e608b504aa0a9feb1144ad9a0b77e; on November 2, each publishable workspace package’s exact npm 0.44.0 endpoint returned HTTP 200, version 0.44.0 and distribution integrity. This is publication evidence, not an inference from source delivery or issue closure. The separate frozen-epoch Phase B gate remains unproven. The November 2 concurrency ruling above selects a conditional direction; the assessed fix-or-revert PR policy is also selected and no rollout is enabled. Phase A children retain their 0.45.0 commitments and recorded individual priorities; the epic itself has no milestone. Structural concurrency is conditionally selected under the November 2 ruling, with all stability, benefit and independence qualifications still required before rollout.

  1. ci: attribute why each failed first-main attempt escaped pre-merge proof #2937 Audit pre-merge escapes in the 49 failed first-main attempts. For each failure family: was the failing test selected for the PR, did it run locally, did hosted confirmation run on the merged head, was confirmation bypassed, or was the failure introduced by an earlier proof-system change? Close attributable escapes with fixes or filed issues.
  2. ci: decide integration lanes for proof-system changes and red-main fixes #2938 Record the settled integration-lane ruling and scope its implementation. Proof-system/global-tooling changes land one at a time under the existing reservation; the next waits for the preceding change’s own main run and the next two main runs to be green. Product work does not wait on that lane. Off-main qualification is required for test optimizations; red-main fixes use the approved expedited path. The deterministic single-file path set defines automatic proof-policy epochs. Expedited proof shows the failing tests red on the bad base and green on the fix head plus affected fast invariants; the next complete main run includes the red set, and a new failure opens a new incident. Preemption reaches only the orchestrator’s own subtree and only to admit a red-main fix. ci: decide integration lanes for proof-system changes and red-main fixes #2938 owns the record and remaining implementation-task briefs; it no longer needs a ruling.
  3. ci: decide whether drafts may request hosted heavy proof instead of local suites #2939 Record the settled hosted-draft-feedback ruling and scope its implementation. A repository script requests affected heavy categories by exact PR/head workflow_dispatch, refusing a changed live head. This is feedback only, without the reservation; ready confirmation remains the only merge proof. Local preflight is check:pr plus focused failing-first/mutation receipts; local long suites are debugging through ci: admit heavy local test commands through a host-wide ledger with priority and preemption #2943 admission. Keep one draft heavy run per PR, cancel its predecessor on push/request, refuse inherited-red work except declared fixes, prioritize ready confirmation, and record runner-slot wait with Epic: CI and integration evidence is collected, archived beyond GitHub retention and reported automatically #2948. The decision record and implementation-task briefs remain; the policy question is answered.
  4. ci: restore one recorded merge path and refuse merges with uncovered local skips #2940 Restore one recorded merge path. Establish why ordinary merges stopped using scripts/integrate.ts after 2026-10-23, then either restore it in the skills or replace it under a recorded ruling. Give the expedited red-main fix a recorded form, so a forced bypass is never the only way out. Local tooling records skipped, timed-out and preempted checks, and the merge path refuses a candidate whose exact-head hosted proof did not cover every selected category.
  5. ci: admit heavy local test commands through a host-wide ledger with priority and preemption #2943 Machine-wide admission for heavy local commands. Sized by the ruling under 3: a host-level limit so concurrent agents queue instead of timing out together, with red-main fixes admitted first. Admission is a durable ledger with priority classes and a preempt command that cancels with cleanup and records preempted, refusing targets outside the caller's orchestrator subtree. The orchestrator drives it; it never holds the state.
  6. ci: report colliding open PRs before claiming and confirming #2941 Detect in-flight collisions before claiming and confirming. A read-only command trial-merges open ready and draft PR heads pairwise (git merge-tree) and compares their affected closures, reporting textual, structural and closure overlap. work-issue consults it before claiming and before confirm. Detection only: no locks between agents.
  7. ci: record a red main centrally, stop doomed proof and file one incident issue #2942 Red-main controller. Complete main results (proof runs even when changeset fails; each main run includes the open red set), a coordinator workflow on main CI completion (strict flake triage, red-main record in the shared ref, one incident issue, cancelling doomed PR runs, clearing on green), a hosted gate that fails doomed proof fast, merge refusal while red, and detection of merges that land anyway.
  8. ci: work a red-main incident through one claimed issue that every orchestrator can add evidence to #2944 Red-main incident workflow in the skills. Orchestrators find the incident from the record, check for duplicates, claim the fix atomically through the record (mirrored to assignment), all collect evidence on the issue, and resume proof when main is green. Blocked by ci: record a red main centrally, stop doomed proof and file one incident issue #2942.

Phase B — concurrency

Gated: starts only after 7 consecutive days within one proof-policy epoch with first-main green at or above 80% and every red incident attributed (ruled on #2938). The November 2 conditional concurrency ruling also requires measured benefit and qualified complete independence/proof/coordination before rollout; it does not declare this gate met.

  1. Measure the current model. Use the recorded work-started/wait-started events and reservation request, admission and refusal events, split by reconfirmation reason (docs/process/ci-throughput.md:78-109). Include runner-slot wait as a first-class metric. Every later task cites this baseline.
  2. Assessed recovery PR on a red-main incident. Detection/incident ownership and causal evidence come from task 7 and the existing incident workflow. Under the November 2 ruling, the owner does a quick bounded assessment and pushes a focused fix PR or targeted revert PR, explaining the choice; either receives normal review and required proof/merge handling. No controller auto-merge or blanket later-squash rollback. Include conflict/uncertain-cause/duplicate and loop-stop cases, actual incident/revert events and exact resulting-main green proof.
  3. Conditionally approved direction, dependent on 9: base-tolerant confirmation and conflicting-closure reservations. Qualify the complete closure/global-input model and deterministic B..B′ comparison, conservative unknown/changed-graph serialization, competing lease races, exact head/tested-base/live-base/squash audit and complete combined-tree main proof under the November 2 ruling. Stack prefix integration still requires separate per-layer qualification (docs/decisions/ci-utility.md:1336-1339). Scope the implementation after 9 reports material benefit; do not start through the stability gate or infer rollout from this policy selection.
  4. Re-measure against 9, matched by cohort under docs/process/ci-throughput.md, comparing first-pass green deliveries, tail latency, runner time, recovery work and post-merge incidents together.

Definition of done

  • Phase A: every merge since the restored path goes through one recorded merge path whose exact-head hosted proof covered every selected category; skipped or timed-out checks are recorded, not silent.
  • Phase A: proof-system changes follow the ruling given under task 2, and the escape audit's attributable families are fixed or filed.
  • A red main is recorded centrally with one incident issue, doomed hosted proof is refused or cancelled, a declared fix can still merge, and merges that land anyway are reported.
  • Open PRs that collide with each other are reported before confirmation, not discovered at merge.
  • Phase B gate met and recorded on this epic: the frozen epoch's dates and its first-main green rate.
  • A detected red-main failure produces a recorded incident and bounded owned assessment, leading to an evidence-backed fix PR or revert PR through normal review, required proof and merge handling. Unsupported attribution remains explicit, rather than forcing a speculative change. Record actual revert events only for reverts, and confirm recovery on the exact resulting main tree; no direct controller auto-merge or silent loss of later good changes.
  • Either two candidates with disjoint affected closures can be confirmed and merged concurrently, or task 9's measurement is recorded showing lease waiting does not justify it.
  • The re-measurement against the task-9 baseline is recorded on this epic, matched by cohort, including post-merge incidents. A faster merge with unfinished main verification does not count (docs/process/ci-throughput.md:138-139).
  • Every policy change made here is recorded in docs/decisions/ci-utility.md with its ruling.

2026-10-25 evidence update: added same-reporter disjoint-cohort measurements, failure-attribution counts and task-scoping implications. Reworded the throughput-ceiling premise as an unmeasured assumption; no policy ruling or implementation is claimed.

2026-10-25 triage at main dcd3b336dbade5e29e8a131eeeb5a0d9cffe60ff: Preserved the concurrent same-reporter cohort measurements and dated evidence supplement, including their attribution/coverage limits. Marked concurrency and automatic rollback as proposed outcomes requiring exact policy/authority rulings, supplied alternatives and an unapproved staged recommendation, and corrected two primary-paper scope statements. Preserve today’s exact-base/global-reservation and proof rules. p2 sorts the wanted unscheduled epic; no child, release commitment, merge permission, policy change or independent verification of the measurement rows is claimed.

2026-10-25 reframe: split the breakdown into Phase A (stabilize proof, integration lanes including a red-main fix lane, one recorded merge path, host admission with orchestrator-driven preemption, collision detection) and a gated Phase B (measure, culprit and revert, conditional base tolerance and partitioning). Added the orchestration and integrator rules to Shared context, excluded an integrator agent, and recorded that practice diverged from the reservation policy after 2026-10-23. Scoped the existing rulings section to Phase B. No policy ruling is claimed; the maintainer selected the phase ordering.

2026-10-25 red-main update: added Phase A tasks 7 (#2942, red-main controller) and 8 (#2944, incident workflow), renumbered Phase B to 9–12 and moved culprit attribution into #2942. Filed and attached Phase A tasks #2937–#2944. Recorded the maintainer's red-main and coordination decisions (#2934 (comment)).

2026-10-25 gate update: Phase B gate set by maintainer ruling on #2938 to 7 consecutive days in one proof-policy epoch, first-main green at or above 80%, every red incident attributed.

2026-10-25 triage at main c670449d3bf1b0639e50ce554d728961f4313add: Folded the later maintainer lane, preemption, expedited-proof, hosted-draft-feedback and amended seven-day gate rulings into the active Phase A breakdown. Preserve the Phase B decision label, exact-base/global reservation rules and separate automatic-revert-merge question; no implementation or policy-record delivery is claimed.

2026-10-26 measurement update: added the separate rolling observation, six new failure reviews, acquisition/attribution denominators and source limitations; recorded the later product attribution of #2900 and aligned measurement-section task references with the current breakdown. Preserved the earlier comparison and existing Phase A/Phase B rulings. No implementation, proof qualification or policy change is claimed.

2026-10-27 evidence update: added the frozen weekly comparison (including slower green delivery despite lower observed runner cost), preceding-week coverage gap, newer rolling cohort and five additional failure reviews. Preserved earlier dated observations and all existing rulings; no proof-policy change, capacity qualification or completed escape audit is claimed.

2026-10-28 measurement update: recorded partial acquisition despite successful workflows, the conservative complete cursor and bounded partial backfill, provisional rolling metrics and two unresolved failed-job observations. Preserved all earlier dated evidence, scope and maintainer rulings; no policy/gate/readiness change or qualified throughput improvement is claimed.

2026-10-29 decision review: the maintainer reaffirmed the existing Phase B deferral and stability threshold. No policy, implementation, qualification or release commitment changed.

2026-10-29 evidence update: added the widened acquisition-completeness lag, bounded partial backfill, cutoff-limited provisional cohort and three unresolved natural failures. Preserved earlier dated evidence, task hierarchy and all maintainer rulings, including the current deferral; no readiness, policy or implementation change is claimed.

2026-10-30 measurement update: recorded recovered recent acquisition completeness, retained historical gaps, updated provisional rolling observations and three new failed-job reviews. Preserved all earlier dated evidence, scope and maintainer rulings; no policy/readiness change or causal throughput qualification is claimed.

2026-10-31 measurement update: added the current overlapping cohort, preserved partial artifacts from failed automation and source-bound fixture attribution alongside unresolved native errors. Preserved earlier dated evidence and all scope, hierarchy and maintainer rulings; no policy, gate or readiness change is claimed.

2026-11-01 measurement update: added conservative coverage lag, bounded partial backfill, above-80% rolling observation without gate qualification, and two unresolved primary-job reviews. Preserved prior dated evidence, scope, task hierarchy and all maintainer rulings; no readiness, policy or gate change.

2026-11-02 source/prerequisite review at main a84d9bfbcb4e608b504aa0a9feb1144ad9a0b77e: verified stable 0.44.0 publication across the nineteen manifest-derived package names and updated Phase A’s satisfied start condition. Refreshed the moved exact-candidate/no-queue/stack/failed-proof policy pointers against the current decision and integrate.ts. Preserve all dated measurements and limitations, native children, 0.45.0 child commitments, exact-base/global reservation, needs: decision and the separate seven-day Phase B ruling. No new metrics, frozen epoch, concurrent integration, automatic revert or task execution is claimed; the epic stays outside agent-ready.

2026-11-02 measurement update: recorded the widened acquisition gap, bounded partial backfill and two newer unresolved browser failures. Preserved prior dated evidence, publication-start-gate reconciliation, hierarchy and maintainer rulings; no policy, readiness or gate change.

2026-11-02 interactive decision review: the maintainer selected A, conditional future concurrency after the unchanged stability gate, measured material benefit and qualified complete independence/coordination/proof controls. Supersedes the October policy deferral only for that direction. Exact-base/global-reservation rules remain binding until the qualified change is delivered. Recovery authority remains the next decision; no rollout, release or proof waiver is claimed.

2026-11-02 recovery decision review: the maintainer refined the options to a quick assessment selecting a focused fix PR or targeted revert PR, normal review/proof/merge handling either way. Updated the proposed outcome, recovery policy, Phase B task 10 and completion evidence accordingly. All decisions in this review are answered; remove needs: decision and add agent-ready under the requested label rule while retaining epic/non-pickable status and every unmet stability/benefit/qualification prerequisite. No recovery execution, merge, rollout or release is claimed.

2026-11-03 measurement update: recorded recovered recent acquisition, remaining timing gaps, expanded failure review and the frozen weekly comparison. Preserved all dated history, native hierarchy and November 2 rulings; no policy/readiness/gate change.

Activity

  1. added
    type: choreTooling, CI, release plumbing, dependencies.
    area: ciCI workflows, the gates they run, release scripts, packaging.
    area: docsThe docs/ tree and the README / AGENTS / CONTRIBUTING surfaces.
    epicA tracking issue. The work lives in its sub-issues; this one is never coded directly.
    on Oct 25, 2026
  2. schlessera commented on Oct 25, 2026

    @schlessera
    OwnerAuthor

    Repository evidence supplement — 2026-10-25

    This supplies the public per-run ledger behind the measurements added to the epic body. Analysis window: 2026-10-21T17:19:18Z through 2026-10-25T08:35:58.244Z. Scope: the 49 failed first automatic main-push attempts, not all PR failures or all failures in the baseline week. The 83 failed primary check jobs break down into 68 unit, 6 runtime_host, 5 browser and 4 pack jobs. A unit job also runs real runtime/browser fixtures; its job name alone does not identify the bug class.

    Provenance and method

    • Reporter source: 6814f2ce5d3ea7f626284ab28b3c230428d06630; normalized snapshot schema version 1.
    • Fixed baseline: 2026-10-14T17:19:18Z–2026-10-21T17:19:18Z. Baseline path/content digest remained 4875c75488cd31195c0a8e2cf56502758116ac4256cae215a03b93a5c5dc81e0 (SHA-256 over sorted relative paths and file bytes).
    • Hosted observation archives include 39920796941, 40072329416, 40174219370, and 40178241202, plus the earlier normalized baseline and complete local acquisition. Selected comparison inputs have no missing/incomplete discovered attempts. The older budget-limited enrichment is retained separately and excluded from this comparison.
    • Each failed primary job log was read successfully. Evidence retains job/run/attempt/source identities, log SHA-256, bounded normalized failure excerpts and issue references. Full logs, environments and credential values were not published.
    • The classifier matched actual assertion/mechanism contexts and reviewed causal investigations. It excluded aggregate cascades, deliberate failing nested controls, successful negative-test stderr and normal retention refusal followed by fresh execution. A timeout or same-source recovery alone was not classified as a test defect.
    • Classification report SHA-256: 92d1299c5a62c7991162614f3e8b85014783ad9694ece3ab7f88a319f913d534. Sixty canonical append-only job annotations were validated through the collector record/report path; they are observational annotations, not passing-proof receipts.

    Reviewed families

    Counts overlap across runs and jobs. documented-repair describes the linked historical repair evidence; it is not a fresh runtime verification by this review.

    Family Runs Jobs Classification Evidence/owner
    sandbox-note-double 13 13 test job log; #2802
    claude-started-abort 11 11 product job log; #2824
    tmpfs-async-refusal 7 7 test job log; #2824
    panel-followup-readiness 5 5 test job log; #2802
    offline-driver-prerequisite 5 5 test job log; #2621
    cross-session-order 4 4 test job log; #2802
    public-reference-guard 4 4 unresolved job log; #2329
    var-socket-walk 4 4 test job log; #2685
    native-index-abort 3 3 unresolved job log; #2921
    node-bun-import 3 3 product job log; #2720
    managed-generation-uid 3 3 test job log; #2720
    browser-destination 2 2 unresolved job log; #2677
    schedule-dst 2 2 test job log; #2721
    hardlink-diagnostic-order 2 2 product job log; #2731
    external-memory-recovery 2 2 unresolved job log
    browser-raster-parity 2 2 unresolved job log; #2469
    nested-worker-diagnostics 2 2 unresolved job log
    apt-mirror-integrity 1 1 infrastructure job log
    child-reaping 1 1 unresolved job log
    unmapped-test-signature 1 1 unresolved job log
    dirty-mount-timers 1 1 unresolved job log; #2894
    registry-tarball-404 1 1 unresolved job log
    owned-loop-ebusy 1 1 unresolved job log; #2420
    route-fallback 1 1 unresolved job log
    continuity-scene 1 1 unresolved job log
    documentation-path 1 1 unresolved job log
    browser-callout-timeout 1 1 unresolved job log
    worker-pipe-release 1 1 unresolved job log
    audit-inventory 1 1 unresolved job log
    hosted-health-budget 1 1 unresolved job log
    installed-tool-inventory 1 1 unresolved job log

    Per-run ledger

    reviewed-classification means every failed primary check job has a reviewed class. partial means at least one class is known but other failed checks remain unresolved. Family lists may include several causes. PR numbers identify the squash association, not a claim that each PR introduced its listed failure.

    pr,run,attempt,attribution,families
    2547,39757842897,1,unresolved,unmapped-test-signature
    2545,39757854465,1,unresolved,audit-inventory
    2609,39758179574,1,partial,offline-driver-prerequisite;public-reference-guard
    2611,39760255890,1,partial,offline-driver-prerequisite;public-reference-guard
    2614,39761735830,1,partial,offline-driver-prerequisite;public-reference-guard
    2618,39764611574,1,partial,offline-driver-prerequisite;public-reference-guard
    2620,39765760155,1,reviewed-classification,offline-driver-prerequisite
    2625,39768275942,1,unresolved,browser-destination
    2613,39773810817,1,unresolved,registry-tarball-404
    2641,39775282043,1,reviewed-classification,apt-mirror-integrity
    2640,39783251973,1,unresolved,browser-destination;owned-loop-ebusy
    2665,39813173706,1,reviewed-classification,var-socket-walk
    2706,39962612189,1,unresolved,route-fallback
    2702,39968267372,1,unresolved,documentation-path;hosted-health-budget
    2717,39975133112,1,reviewed-classification,managed-generation-uid;node-bun-import;schedule-dst
    2719,39977613080,1,reviewed-classification,managed-generation-uid;node-bun-import;schedule-dst
    2722,39985504099,1,partial,browser-raster-parity;hardlink-diagnostic-order;managed-generation-uid;node-bun-import
    2728,39991237120,1,reviewed-classification,hardlink-diagnostic-order
    2724,40026244740,1,reviewed-classification,var-socket-walk
    2742,40034416635,1,reviewed-classification,var-socket-walk
    2740,40037435256,1,reviewed-classification,var-socket-walk
    2750,40040648796,1,unresolved,nested-worker-diagnostics
    2791,40071933486,1,reviewed-classification,cross-session-order;sandbox-note-double
    2794,40073295502,1,reviewed-classification,cross-session-order;panel-followup-readiness
    2730,40073397871,1,reviewed-classification,panel-followup-readiness;sandbox-note-double
    2743,40078564661,1,reviewed-classification,cross-session-order;sandbox-note-double
    2797,40081701647,1,reviewed-classification,sandbox-note-double
    2808,40082818803,1,partial,native-index-abort;panel-followup-readiness;sandbox-note-double
    2812,40085754409,1,reviewed-classification,sandbox-note-double
    2798,40085808179,1,reviewed-classification,sandbox-note-double
    2818,40089158841,1,reviewed-classification,cross-session-order
    2817,40089165767,1,partial,nested-worker-diagnostics;panel-followup-readiness;sandbox-note-double
    2804,40090671524,1,reviewed-classification,claude-started-abort;panel-followup-readiness;sandbox-note-double;tmpfs-async-refusal
    2825,40093715370,1,reviewed-classification,claude-started-abort;sandbox-note-double;tmpfs-async-refusal
    2822,40093722720,1,partial,child-reaping;claude-started-abort
    2827,40095940441,1,reviewed-classification,claude-started-abort;sandbox-note-double;tmpfs-async-refusal
    2823,40098389182,1,reviewed-classification,claude-started-abort;sandbox-note-double;tmpfs-async-refusal
    2829,40105332240,1,reviewed-classification,claude-started-abort;sandbox-note-double;tmpfs-async-refusal
    2810,40105370967,1,reviewed-classification,claude-started-abort
    2813,40105420585,1,partial,browser-callout-timeout;browser-raster-parity;claude-started-abort;installed-tool-inventory
    2839,40109399392,1,reviewed-classification,claude-started-abort
    2838,40109765507,1,reviewed-classification,claude-started-abort;tmpfs-async-refusal
    2841,40111962805,1,reviewed-classification,claude-started-abort;tmpfs-async-refusal
    2861,40140747836,1,unresolved,worker-pipe-release
    2885,40157111874,1,unresolved,external-memory-recovery
    2882,40157659182,1,unresolved,native-index-abort
    2886,40157912094,1,unresolved,dirty-mount-timers;native-index-abort
    2890,40166236508,1,unresolved,external-memory-recovery
    2884,40175510636,1,unresolved,continuity-scene

    Measurement limits

    26/49 attempts have complete reviewed attribution, 9/49 partial, 14/49 unresolved. Product/test/infrastructure occurrence counts overlap (15/29/1 respectively). The reason each failure escaped pre-merge proof has not been established. Missing incident surveillance is unknown, not zero incidents. Current live-main health cannot be inferred from these historical counts. No automatic revert, proof-policy change or new runtime reproduction was performed.

    Daily collection now carries the annotations and bounded review of new failed main jobs; weekly review retains full/partial/unresolved attribution separately. Actual worker work/wait telemetry and incident events remain prerequisites for capacity and production reliability claims.

  3. added
    priority: p2Wanted, not scheduled. Pulled in when a milestone has room.
    needs: decisionBlocked on a maintainer or designer ruling, not on code.
    on Oct 25, 2026
  4. schlessera commented on Oct 25, 2026

    @schlessera
    OwnerAuthor

    2026-10-25 triage at main dcd3b336dbade5e29e8a131eeeb5a0d9cffe60ff: Preserved the concurrent same-reporter cohort measurements and dated evidence supplement, including their attribution/coverage limits. Marked concurrency and automatic rollback as proposed outcomes requiring exact policy/authority rulings, supplied alternatives and an unapproved staged recommendation, and corrected two primary-paper scope statements. Preserve today’s exact-base/global-reservation and proof rules. p2 sorts the wanted unscheduled epic; no child, release commitment, merge permission, policy change or independent verification of the measurement rows is claimed.

  5. schlessera commented on Oct 25, 2026

    @schlessera
    OwnerAuthor

    Epic state — 2026-10-25: reframed into two phases

    The 2026-10-25 measurements, together with repository history since 2026-10-14, change what this epic should do first. The breakdown in the body has been rewritten to match; this comment records why.

    What the evidence says

    • The repository-wide lease is not the measured constraint. Observed reservation occupancy was 5.52% (baseline) and 4.22% (subsequent cohort). The busiest lease day, 2026-10-20, had 21 acquisitions at a median hold of 21 minutes (p90 23, max 39, n=44 complete holds), about 31% of the day. No cohort ran the lease saturated.
    • Ordinary merges have not gone through the reservation since 2026-10-23 12:43 UTC+2. refs/heads/brain-integration-reservation records no acquisition after that, while roughly 100 squashes landed on main. scripts/integrate.ts did not change in that window. The cohort's 3-second median ready → merge is consistent with merges that never waited on a confirmation. The body's premise that "every ordinary confirmation and merge goes through one reservation" therefore describes the policy, not current practice.
    • First-main green was about half in both cohorts (42.4% baseline, 54.2% subsequent). Red main predates the lease bypass and is not caused by it.
    • Recent work is concentrated in one area. Of 177 squashes since 2026-10-18, 88 touched packages/ui-server, 57 packages/ui-sdk, 31 packages/ui-react, 23 packages/ui-backend-claude and 21 packages/ui-backend-pi; the 13 remaining packages share 20. Changes in that area select whole runtime, browser, native and integrated categories, so two in-flight candidates almost never have disjoint closures. Closure partitioning (old tasks 3–4) would rarely admit concurrency on this workload.
    • About a quarter of recent squashes changed the proof system itself. By subject, 95 of 413 squashes since 2026-10-14 were CI, test-harness, sharding, timeout, deadline or watchdog changes (17 of 26 on 2026-10-20 and 10-21); 49 touched workflows, scripts/test.ts, scripts/unit-*, scripts/ci-*, bunfig.toml or scripts/check-pr.ts. These changes alter every candidate's proof at once, contribute to red main after merge, and mix proof-policy epochs inside both cohorts, so neither cohort is a usable baseline.
    • A red main has no fast exit. A recent fix for red main waited behind unrelated PRs for local proof resources until the maintainer had it merged with every check bypassed. Candidates queued ahead of it were proving against an already red base.
    • Local proof does not scale with agents. Low-level changes in the current release need system-wide tests. Long suites run concurrently on one host choke it and time out, and agents then skip checks to get past the timeouts. A skipped check that later fails on main is a pre-merge escape.

    These counts are subject- and path-based approximations from git log --first-parent origin/main; they bound the effect rather than attribute each run.

    What changes

    The epic proceeds in two phases. The maintainer selected this ordering on 2026-10-25; it is an ordering of work, not a policy ruling.

    • Phase A — stabilize proof and restore one merge path. No structural concurrency change. It starts after the current release's backend work lands, when changes are expected to become feature-driven again.
    • Phase B — concurrency. It is gated: it starts only after a frozen proof-policy epoch of about two weeks with first-main green above an agreed threshold. Then the current model is measured, culprit attribution and revert are built, and base tolerance or closure partitioning are pursued only if measured lease waiting justifies them.

    How agents are organized

    Several people each run an orchestrator agent that manages several workers. The structure follows the hierarchy that worked in the Cursor report, without the integrator role it dropped:

    • Command is a tree. Maintainer, then orchestrators, then workers. A worker takes commands from exactly one orchestrator; an orchestrator acts only within its own subtree. Only the maintainer's revoke reaches across subtrees directly.
    • Shared resources are ledgers, not authorities. Host slots, the reservation and a red-main record are arbitrated by a CLI under fixed rules. Restrictions only add up: a worker proceeds only when its orchestrator and every ledger allow it. Cross-subtree contention is settled by ledger priority, never by one orchestrator directing another's worker.
    • The orchestrator decides; commands execute. Admission, priority, preemption and the red-main response go through deterministic commands with durable state. An orchestrator that crashes or loses context leaves nothing inconsistent, and a worker without one degrades to plain admission limits, never to no limit.
    • Preemption cancels with cleanup. A preempted run is recorded as preempted: rerun later, never counted as checked, never treated as a failure.
    • The integrator is a service, not an agent. scripts/integrate.ts grows into it: lanes, priority, admission, red-main record, proof checks, squash and revert. Workers own their PRs end to end, including conflicts and fixing what they break. Cursor removed its integrator agent because it "created more bottlenecks than it solved" (https://cursor.com/blog/scaling-agents). Anthropic describes no merger agent at 30,000+ merges a day, only a queue-health monitor (https://www.anthropic.com/engineering/redesigning-source-control). An optional watcher may report main's health to the orchestrator, but it never merges.

    Two lanes are proposed, both needing a maintainer ruling filed as one task. A red-main fix lane gives a declared fix precedence under a defined expedited proof, and pauses ordinary confirmations that overlap the failing category. The other reverses part of the original premise and serializes more for a small class of changes: proof-system and global-tooling changes would land one at a time, each followed by a stability window on main, while product changes flow.

  6. 26 remaining items

  7. schlessera commented on Nov 2, 2026

    @schlessera
    OwnerAuthor

    Decision (maintainer, 2026-11-02): selected A — conditionally approve concurrency. Permit the future disjoint-base/closure-reservation direction only after (1) seven consecutive days in one proof-policy epoch, first-main green at least 80% and every red incident attributed, (2) a complete measured baseline showing material lease/reconfirmation benefit, and (3) qualification of complete affected closures/global inputs, conservative serialization for unknown/changed graphs or conflicts, deterministic coordination, exact head/base/squash audit and full combined-tree main proof/incident ownership. Preserve selected proof, existing red-main controls and permissions; update the CI decision and concrete native child briefs before implementation.

    This supersedes the October concurrency deferral as a conditional policy direction, not a finding that any gate is met or rollout can start. Current exact-base confirmation and the global reservation still apply. All dated evidence, limitations, native children and release commitments are preserved. Automatic recovery/revert-merge authority is the remaining question, so keep needs: decision for now. No implementation, merge, release or proof waiver occurs in this review.

  8. added
    agent-readyContext and acceptance criteria are complete enough for an agent to take this unattended.
    and removed
    needs: decisionBlocked on a maintainer or designer ruling, not on code.
    on Nov 2, 2026
  9. schlessera commented on Nov 2, 2026

    @schlessera
    OwnerAuthor

    Decision (maintainer, 2026-11-02): do a quick assessment and decide between pushing a fix PR or a revert PR; let either go through normal review.

    The incident owner uses causal/current-tree evidence to choose a small promptly provable fix, or a targeted revert when it is the safer/quicker supported recovery. Record the rationale; a red check/latest squash alone is not proof of cause. Assess dependencies/conflicts, preserve unrelated/later good work, and leave uncertainty explicit when neither change is supportable. Either PR follows normal required review, proof and recorded merge handling (including the approved declared-red-main-fix route where applicable), with exact resulting-main recovery proof. No direct controller auto-merge or unbounded recovery loop.

    This replaces the proposed revert-only/automatic-revert-merge outcome. Updated the current body, native task-10 brief and completion evidence. Conditional concurrency remains selected with all gates and current exact-base/global-reservation policy intact until qualified rollout. All policy decisions are now recorded: remove needs: decision; with no other needs label, add agent-ready per the requested labeling rule. It remains a non-pickable epic whose scoped children own implementation; no gate passage, actual recovery, implementation, merge or release is claimed.

  10. schlessera commented on Nov 3, 2026

    @schlessera
    OwnerAuthor

    Daily evidence — 2026-11-03 (UTC)

    Recent acquisition recovered: 24 new successful workflow artifacts preserved, 2 completed / 22 partial manifests. The latest validated archive 41540323232 extends overlapping creation-window coverage through 2026-11-02T22:33:36.688Z, closing the previous gap. It contains 907 attempts / 904 runs / 5,161 jobs, no missing/incomplete inventories or lifecycle/reservation errors, but 14 observed jobs still lack duration. Complete inventories are not complete timings. Newer partial evidence reaches 2026-11-03T07:33:33.411Z, leaving roughly nine hours unvalidated. A 30-request resource/reservation backfill of 2026-10-31T22:33:36.688Z–2026-11-03T07:33:33.411Z remained partial despite exit 0 and did not advance the cursor.

    Rolling delivery interval 2026-10-29T23:33:28.071Z–2026-11-02T22:33:36.688Z: 60 PRs, 51 first-main green / 9 failed / 0 cancelled / 0 unknown (85%). Failed attribution 2 full / 0 partial / 7 unresolved / 0 unreviewed. This overlaps earlier rolling cohorts and spans changing proof policy; 85% is not qualification of the frozen seven-day gate. Receipt/squash comparisons are 60 unknown. Joined history retains 4 incomplete snapshots, 200 resource problems, 6 unattributed attempts and 19 lifecycle errors, alongside the 14 missing durations.

    Five new failed primary jobs across four first-main attempts:

    The last two attempts are after the primary analysis cutoff, included only in fresher/cumulative review. Five logs acquired successfully, at most two concurrent requests, no known log backlog. Two job-specific annotations added; 71 total. Cumulative 78 failed attempts / 113 primary jobs: 36 full / 10 partial / 32 unresolved. Successful negative controls and retained-proof refusal followed by fresh execution were excluded; no fresh reproduction was run here. Explicit CI incident/recovery evidence (#3361/#3419) exists, but this is not complete product-incident surveillance. The separate #2937 escape audit concerns the original 49, not automatic escape attribution for all 78.

    Newly available frozen weekly findings: 2026-10-24T23:33:28.071Z–2026-10-31T23:33:28.071Z, versus 2026-10-17T23:33:28.071Z–2026-10-24T23:33:28.071Z. Latest 117 deliveries, 87/116 known first-main green (75%), one unknown; preceding 156, 85/146 (58.2%), ten unknown. Green deliveries/day 12.14→12.43 (+2.4%), observed runner-minutes/green 422.94→396.79 (−6.2%); create→green median 114.79→111.45min, p90 51.69→12.71h (n=86/93). Against the fixed October 14–21 baseline, green deliveries/day remain 24.3% lower, despite 46.6% lower observed runner-minutes/green. The 30%-fail/two-extra-push model gives 2.6 full-run equivalents and 27.3% estimated compute savings using four in-window exhaustive references, not causal wall-time savings; the original baseline had one reference. Two week delivery cohorts are disjoint; fixed baseline overlaps the preceding week. Mixed workload/runtime/policy, unknown outcomes and incomplete evidence prevent a causal parallel-throughput claim. The review's reporter was d824fc8, schema 1; corrected estimates exclude the raw reporter's after-cutoff reference pool.

    Daily reporter: trusted default branch e04768a, schema 1, Bun 1.4.2 (collector source). Baseline/prior reports preserved. Actual worker activity, caller wait, overall incident rates and independent proof guarantees remain unknown. The newly recorded conditional-concurrency and normally reviewed fix/revert rulings are preserved; measurement changes no gate, policy, readiness or hierarchy.

  11. removed
    agent-readyContext and acceptance criteria are complete enough for an agent to take this unattended.
    on Nov 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area: ciCI workflows, the gates they run, release scripts, packaging.area: docsThe docs/ tree and the README / AGENTS / CONTRIBUTING surfaces.epicA tracking issue. The work lives in its sub-issues; this one is never coded directly.priority: p2Wanted, not scheduled. Pulled in when a milestone has room.type: choreTooling, CI, release plumbing, dependencies.

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions