Skip to content

ci: decide whether drafts may request hosted heavy proof instead of local suites #2939

Closed

Description

@schlessera

Part of #2934 (Phase A, task 3). All questions were ruled on 2026-10-25 (see the decision comment); what remains is recording the ruling and filing its implementation tasks. It amends the 2026-10-14 ruling in #1849/#1854.

Context

Changes in the current release are mostly low-level: of 177 squashes since 2026-10-18, 88 touched packages/ui-server and 57 packages/ui-sdk, and changes there need system-wide runtime, native and browser proof. Today an agent has two ways to run that proof:

  • Locally. Several agents on one host run long suites concurrently. The host chokes, suites hit their timeouts, and agents skip the checks to proceed. Individual native fixtures take 130–150 s per file on hosted runners (for example scripts/native-proof/hosted-app.test.ts). Overlapping runs in one workspace also serialize on exclusive workspace output access.
  • Hosted, through ready-candidate confirmation. Drafts run cheap gates only, and complete affected proof runs only when a PR is marked ready (docs/decisions/ci-utility.md:472-479). Under policy, that requires holding the repository-wide reservation (docs/decisions/ci-utility.md:1237-1246), so it is a final confirmation, not an iteration tool.

Neither scales with agent count. Standard runner minutes are free (docs/decisions/ci-utility.md:3-10), but account runner concurrency is not unlimited (docs/decisions/ci-utility.md:1192).

Questions for the ruling

  1. May an agent request the PR's affected heavy categories on a draft head, without the reservation, as feedback that never counts as proof?
  2. How is it requested: a workflow_dispatch with the PR number and head SHA, a label event, or a comment? It must bind to one exact head, like the existing request (docs/decisions/ci-utility.md:480).
  3. Does local preflight then formally stop running long suites, keeping only check:pr and focused failing-first or mutation receipts?
  4. What protects account runner concurrency: a per-PR cap on draft heavy runs, cancellation of a PR's earlier draft heavy run on a new push, or both?
  5. Is runner-slot wait recorded as a first-class metric in scripts/ci-timings.ts, so the next ceiling is visible?

A documented "no" closes this issue, with the reason recorded and an alternative for local contention (Phase A task 5) chosen instead.

Out of scope

  • Changing what counts as merge evidence: exact-head ready confirmation remains the only proof.
  • Implementing the request path; it becomes its own task once ruled.

Acceptance criteria

Verification

The amendment states the runner-concurrency protection and how it is measured.

Contract impact

None.

2026-10-25: all questions ruled by the maintainer; acceptance criteria now cover recording the ruling and filing its implementation tasks.

Activity

  1. added
    type: choreTooling, CI, release plumbing, dependencies.
    area: ciCI workflows, the gates they run, release scripts, packaging.
    needs: decisionBlocked on a maintainer or designer ruling, not on code.
    on Oct 25, 2026
  2. schlessera commented on Oct 25, 2026

    @schlessera
    OwnerAuthor

    Decision (maintainer, 2026-10-25): hosted heavy proof on draft heads.

    1. Yes, as feedback only. An agent may request its PR's affected heavy categories on a draft head without holding the reservation. The result never counts as proof; exact-head ready confirmation remains the only merge evidence. This amends the Epic: tests give faster feedback with less redundant work #1849/ci: separate fast feedback, final confirmation and parallel endurance suites #1854 rule that drafts run cheap gates only (docs/decisions/ci-utility.md:472-479).
    2. Request by workflow_dispatch with the PR number and head SHA, through a repository script. The workflow refuses when the live PR head differs from the requested SHA.
    3. Local long suites are for debugging only. Local preflight is check:pr plus focused failing-first and mutation receipts. Long suites run locally only to debug a specific failure, through host admission (ci: admit heavy local test commands through a host-wide ledger with priority and preemption #2943).
    4. Runner concurrency protection: at most one draft heavy run in flight per PR; a new push or request cancels that PR's earlier one; refused while main is red unless the PR is a declared fix (ci: record a red main centrally, stop doomed proof and file one incident issue #2942); ready confirmations take priority.
    5. Runner-slot wait (job created to job started) becomes a first-class metric, per workflow and category, in scripts/ci-timings.ts and the Epic: CI and integration evidence is collected, archived beyond GitHub retention and reported automatically #2948 reports.
  3. added
    agent-readyContext and acceptance criteria are complete enough for an agent to take this unattended.
    and removed
    needs: decisionBlocked on a maintainer or designer ruling, not on code.
    on Oct 25, 2026
  4. added this to the 0.45.0 milestone on Oct 25, 2026
  5. added
    priority: p2Wanted, not scheduled. Pulled in when a milestone has room.
    on Oct 25, 2026
  6. self-assigned this
    on Nov 2, 2026
  7. schlessera commented on Nov 2, 2026

    @schlessera
    OwnerAuthor

    Claimed as part of working epic #2934 end to end (claim token wi-20261102-fd2d2f21). Order chosen for earliest queue-velocity benefit: #2938+#2939 rulings first (they unlock hosted draft feedback), then the red-main/merge-path chain (#2942, #2940, #2944); #2943, #2941 and #2937 run in parallel since they touch disjoint code. Intended branch: docs/2938-2939-record-integration-rulings (shared with #2938).

  8. schlessera commented on Nov 2, 2026

    @schlessera
    OwnerAuthor

    Delivered by #3274, squash 6dfdafe9ecdc3cd36388a0c445d928c3fdb4339f. Its parent is the tested base d6426a76 and its tree is 99a6248e, the same tree as candidate run 41420146145. The ruling is recorded in docs/decisions/ci-utility.md under "Integration lanes, red-main fixes and hosted draft feedback". Implementation tasks are filed and attached to #2934: #3268, #3269, #3270, #3271 (lane, from #2938) and #3272, #3273 (draft feedback, from #2939; runner-slot wait remains #2952). The main-push run 41420740208 was still running when this was posted.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

agent-readyContext and acceptance criteria are complete enough for an agent to take this unattended.area: ciCI workflows, the gates they run, release scripts, packaging.priority: p2Wanted, not scheduled. Pulled in when a milestone has room.type: choreTooling, CI, release plumbing, dependencies.

Projects

No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions