Skip to content
CI

chore(e2e): review screenshots for #2703 #11242

Sign in to view logs

chore(e2e): review screenshots for #2703

chore(e2e): review screenshots for #2703 #11242

Workflow file for this run

.github/workflows/ci.yml at 0570b61
name: CI
on:
push:
# `main` is the default branch and the integration target: day-to-day PRs
# land there under the LIGHT tier. `release` only ever receives promotion
# PRs from `main`, and those run the FULL tier — so `release` is always in a
# state a release can be cut from. See the heavy-tier guards on `bench` /
# `e2e` below, and docs/testing-strategy.md.
branches: [release, main]
# A release tag must receive this exact same aggregate gate before the
# publishing workflow will package it. See release-mac.yml's preflight.
tags: ['v*']
# `labeled` is included so adding the `update-screenshots` or `ci-full` label
# kicks a fresh run on the PR head — plan-e2e then forces mode=full
# (`update-screenshots` additionally renders every reference shot into an
# immutable candidate artifact). Without it the label only takes effect on the
# next push/sync, so a label-only request would never regenerate. (For
# pull_request events GitHub reads these triggers from the BASE branch's
# workflow, so this is only active once merged to main.)
pull_request:
# `ready_for_review` is required because the heavy tier (build / e2e / bench)
# is gated on `draft == false` below: without it, promoting a draft would
# never dispatch those jobs and `CI Passed` would sit on the draft's
# heavy-jobs-skipped result forever.
types: [opened, synchronize, reopened, labeled, ready_for_review]
# No `branches:` filter — PRs into `main` and the promotion PRs into
# `release` both run this workflow. Which TIER they get is decided per job by
# `github.base_ref`, not here, so a PR retargeted from `main` to `release`
# picks up the full tier on its next sync without touching this file.
# Nightly full e2e on GitHub-hosted runners — the comprehensive safety net.
# Merge-eligible PRs run either the oracle-selected subset or, when the oracle
# cannot safely thin, the full suite. Main pushes remain cheap because their
# exact commit was already gated as a PR; the nightly catches direct-push and
# environment drift. Off-:00 minute to avoid the cron stampede.
schedule:
- cron: '17 6 * * *'
# NOTE: there is deliberately no `merge_group` trigger. GitHub's merge queue
# requires Enterprise Cloud for private repositories and this org is on Team,
# so the trigger could never fire. The `main` -> `release` promotion flow is
# the stand-in: `main` absorbs the day's merges under the light tier, and
# the promotion PR runs the full tier once for the whole batch. What we give
# up versus a real queue is automatic bisection — a red promotion names a
# batch, not a commit — which is why promotions should be frequent enough to
# stay small.
concurrency:
group: ci-${{ github.event.pull_request.number || github.ref }}
# Nightly schedule must not cancel an in-flight tip push/PR: cron can fire
# hours late, and schedule+push share `ci-refs/heads/main`. They also use
# different e2e runners (hosted vs self-hosted), so running in parallel is
# fine. A new push/PR sync still cancels older runs in the group.
cancel-in-progress: ${{ github.event_name != 'schedule' }}
jobs:
# Job timeouts: every job below sets `timeout-minutes`, and
# scripts/ci-workflow-invariants.test.ts fails the build if a new one forgets.
# GitHub's default is 360 minutes, which is survivable on hosted runners (you
# pay for it) and is not on this fleet: runners are EPHEMERAL and serve both
# tiers, so one wedged job parks a whole runner — a sizeable slice of total
# capacity — for six hours while every other PR queues behind it. Observed on
# a `main` push that sat at 167 minutes with no job-level cap to stop it.
#
# The caps are leak-stoppers, NOT latency SLAs. Each is roughly 3-5x the
# observed duration so that a slow-but-healthy run (cold dependency cache, a
# lint OOM retry, a saturated box) never goes red on the clock — a false red
# costs more than the leak it would prevent. Queue time is excluded: Actions
# starts this clock when the job begins executing, not when it is dispatched.
# Tighten only with data (a run record showing the real p99), and raise
# rather than let a genuinely-slower job flake.
#
# Check-tier runner routing: GitHub-hosted is the DEFAULT, and the org
# self-hosted fleet is opt-in. Every `runs-on` below (except e2e, which routes
# by label with its own nightly-hosted exception, and release-mac in its own
# workflow) reads the SELF_HOSTED_CHECKS Actions variable:
# (unset or '') -> ubuntu-latest <- default
# SELF_HOSTED_CHECKS=copse-checks -> the fleet (ci-runners/)
#
# This inverts the old CHECKS_RUNNER scheme, which defaulted to whatever a
# variable said and treated hosted as the fallback. Two things changed when
# this repository went public:
# 1. Standard GitHub-hosted minutes are free and unlimited on public repos,
# so the cost argument that justified the fleet is gone.
# 2. Public repos get 4-vCPU / 16 GiB hosted runners — more than double the
# 6 GB the self-hosted check cgroup allows. Hosted is now the BIGGER box,
# not the fallback one.
# The rename is load-bearing, not cosmetic: `vars.X` resolves repo-then-org,
# so an org-level CHECKS_RUNNER=copse-checks used to re-route this repo's
# whole check tier with no change here and no signal on the PR. Reading a name
# that is set nowhere makes hosted the default in code rather than in a
# variable someone else can flip.
#
# Opting a tier back onto the fleet is therefore deliberate:
# gh variable set SELF_HOSTED_CHECKS --repo copse-dev/agent-pane --body copse-checks
# gh variable delete SELF_HOSTED_CHECKS --repo copse-dev/agent-pane
# (or Settings -> Secrets and variables -> Actions -> Variables.) Nothing
# auto-detects the fleet going offline, so if you set it and the pool has no
# online runner, routed jobs queue until a runner returns or the variable is
# cleared — clear it when taking the fleet down.
#
# Fork PRs run only the `precheck` safe tier, forced onto a GitHub-hosted
# runner with a read-only token. Every heavier job that executes repository
# code keeps its same-repo `if` guard, so untrusted commits never run on the
# self-hosted fleet.
# Cheap static gate + e2e planning. Runs the fast static checks (typecheck,
# lint, format, dead-code, oracle liveness) AND computes the e2e oracle plan in
# a single job. Everything expensive — coverage, build, e2e — hangs off this
# via `needs: precheck`, so a typo, lint error, or bad format short-circuits
# the pipeline before it burns runner minutes on the build + e2e shards.
# Folding the former standalone `plan-e2e` job in here drops a duplicate
# checkout + node setup (and its rounded-up billed minute) from every run.
precheck:
# GitHub-hosted by default; the self-hosted check fleet is opt-in via
# SELF_HOSTED_CHECKS (see the routing note above).
# History: the 6 GB self-hosted cgroup has cgroup-killed `tsc` / `eslint .`
# under load (exit 137), which is why the steps below retry on 137. On the
# public-repo hosted runner (16 GiB) that headroom is no longer scarce; the
# retries stay because opting back onto the fleet must not silently regress.
#
# Fork PRs are pinned to GitHub-hosted: this is the one job that executes
# fork code (typecheck/lint run the PR's package scripts and plugins), and
# it must never reach the self-hosted fleet — same invariant gitleaks.yml
# states for its standalone fork scan. Hosted minutes are free once the
# repository is public, so this costs nothing on the runs it applies to.
runs-on: ${{ (github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name != github.repository) && 'ubuntu-latest' || vars.SELF_HOSTED_CHECKS || 'ubuntu-latest' }}
timeout-minutes: 30
permissions:
contents: read
# Consumed by the e2e job: how much of the e2e tier this run needs (mode/
# specs), and the runner-dependent shard fan-out (e2e_shards / shard_total).
outputs:
mode: ${{ steps.plan.outputs.mode }}
specs: ${{ steps.plan.outputs.specs }}
e2e_shards: ${{ steps.shards.outputs.list }}
e2e_shard_total: ${{ steps.shards.outputs.total }}
# How much of the UNIT tier the `check` job below needs. Empty is read as
# `full` there, so a branch of the plan step that forgets to set it fails
# safe rather than silently thinning the suite.
unit_mode: ${{ steps.plan.outputs.unit_mode }}
unit_specs: ${{ steps.plan.outputs.unit_specs }}
steps:
# fetch-depth: 0 so the e2e oracle (plan step) can diff against the PR base
# / previous main tip.
- uses: actions/checkout@v7.0.1
with:
fetch-depth: 0
# Do not leave even the read-only GITHUB_TOKEN in a fork checkout.
# Same-repo runs keep credentials for the oracle's base ref fetches.
persist-credentials: ${{ github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name == github.repository }}
# Same-repository PRs already execute inside this job's check-fleet trust
# boundary. Scan them here so secret detection blocks the required
# `CI Passed` gate without dispatching a second GitHub-hosted job. Forks
# remain on the standalone hosted workflow in gitleaks.yml.
- name: Install pinned gitleaks CLI
if: github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name == github.repository
env:
GITLEAKS_VERSION: 8.30.1
GITLEAKS_LINUX_X64_SHA256: 551f6fc83ea457d62a0d98237cbad105af8d557003051f41f3e7ca7b3f2470eb
shell: bash
run: |
set -euo pipefail
archive="gitleaks_${GITLEAKS_VERSION}_linux_x64.tar.gz"
path="${RUNNER_TEMP}/${archive}"
curl --fail --location --retry 3 \
--output "${path}" \
"https://github.com/gitleaks/gitleaks/releases/download/v${GITLEAKS_VERSION}/${archive}"
echo "${GITLEAKS_LINUX_X64_SHA256} ${path}" | sha256sum --check --strict
tar -xzf "${path}" -C "${RUNNER_TEMP}" gitleaks
"${RUNNER_TEMP}/gitleaks" version
- name: Scan repository history for secrets
if: github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name == github.repository
run: |
"${RUNNER_TEMP}/gitleaks" git --redact --verbose --no-banner \
--log-opts="--full-history --diff-filter=tuxdb HEAD" .
- uses: ./.github/actions/setup
# Retry once on exit 137 (SIGKILL / cgroup OOM). The check fleet caps each
# runner at ~6 GB; under load, tsc/eslint can get killed even though a
# second attempt on a quieter container succeeds. Real type/lint errors
# exit non-137 and still fail the gate immediately.
- name: Typecheck
run: |
set +e
npm run typecheck
status=$?
set -e
if [ "$status" -eq 0 ]; then exit 0; fi
if [ "$status" -ne 137 ]; then exit "$status"; fi
echo "::warning::typecheck OOM-killed (137); retrying once"
sleep 10
npm run typecheck
- name: Lint
run: |
# Typecheck just exited; give the cgroup a beat to reclaim before
# lint starts, then retry a couple of times on 137. The lint script
# runs typed projects sequentially so their programs do not share one
# V8 heap; a retry is still useful for external cgroup pressure.
sleep 15
attempt=1
while true; do
set +e
npm run lint
status=$?
set -e
if [ "$status" -eq 0 ]; then exit 0; fi
if [ "$status" -ne 137 ] || [ "$attempt" -ge 3 ]; then exit "$status"; fi
echo "::warning::lint OOM-killed (137); retry $attempt/3"
attempt=$((attempt + 1))
sleep 20
done
- run: npm run format:check
- run: npm run check:dead-code
# Enforce the API protocol's compatibility rule (docs/api-protocol.md).
# The unit test only pins that the committed manifest matches the sources,
# which a breaking change satisfies by regenerating; this is the step that
# reads the change against the base and fails a breaking one that did not
# bump API_PROTOCOL_VERSION. Only meaningful with a base to compare to,
# so it is skipped on schedule / branch-create runs.
- name: API protocol compatibility
env:
# PR base on pull_request; previous tip on push. Empty for schedule.
BASE_SHA: ${{ github.event.pull_request.base.sha || github.event.before }}
run: |
set -euo pipefail
# No usable base (schedule, first push, branch create) -> nothing to
# compare against. Skipping here is safe: the PR that introduces a
# change always has one.
if [ -z "${BASE_SHA:-}" ] || [ "$BASE_SHA" = "0000000000000000000000000000000000000000" ]; then
echo "No base sha for ${GITHUB_EVENT_NAME} -> skipping the API protocol comparison"
exit 0
fi
# A base that cannot be read fails the step rather than skipping it:
# a compatibility gate that passes whenever CI cannot see the base is
# worse than no gate. The script distinguishes that from a base that
# genuinely predates the protocol, which it allows.
git fetch --no-tags --quiet origin "$BASE_SHA" 2>/dev/null || true
node scripts/gen-api-protocol.mts --compare-ref "$BASE_SHA"
# Fail closed on high/critical dependency advisories. This runs after the
# lockfile-exact setup, so it audits precisely the tree CI will execute.
# The registry's audit endpoint intermittently times out from hosted
# runners (2026-09-04: three attempts in a row here while unrelated
# branches passed either side of them), and pnpm's own two retries span
# too short a window. Retry only when the registry could not be reached,
# so a real advisory still fails on the first attempt.
- run: |
attempt=1
while true; do
set +e
pnpm audit --audit-level=high 2>&1 | tee "$RUNNER_TEMP/pnpm-audit.log"
status=${PIPESTATUS[0]}
set -e
if [ "$status" -eq 0 ]; then exit 0; fi
if ! grep -qE 'ERR_SOCKET_TIMEOUT|FetchError|ECONNRESET|ETIMEDOUT' "$RUNNER_TEMP/pnpm-audit.log" || [ "$attempt" -ge 3 ]; then
exit "$status"
fi
echo "::warning::pnpm audit could not reach the registry; retry $attempt/3"
attempt=$((attempt + 1))
sleep 60
done
# Guard the e2e oracle that gates the `e2e` job: every spec must stay
# selectable (liveness) and the mapping invariants must hold, so the gate
# can't silently start skipping specs it should run.
- run: npm run check:oracle
# Test oracle (scripts/test-oracle.mts) decides how much of the e2e tier
# this run needs. On a pull_request it maps the diff (vs the PR base) to the
# specs it can affect; on a push to main it maps vs the previous main
# commit. The nightly `schedule` always runs the full suite. The output
# drives the `e2e` job below:
# mode=full → all 8 shards run the whole CI suite
# mode=subset → only the listed specs run, distributed across the shards
# mode=skip → nothing e2e-relevant changed; shards report green, no run
# subset is used only for confident, UI-coupled diffs; LOW-confidence
# (backend logic with no selector coupling) and broad changes fall back to
# full. Full coverage on every push is no longer guaranteed — the nightly
# is the net.
#
# The same call also emits a UNIT plan (`unit_mode` / `unit_specs`) that the
# `check` job below applies, but ONLY to a PR stacked on another PR's branch
# — see that job for why. Every early-exit branch here predates the oracle's
# analysis of the diff, so each one emits `unit_mode=full`: only the oracle
# path has looked at what actually changed, so only it may thin.
- id: plan
env:
EVENT: ${{ github.event_name }}
BASE_REF: ${{ github.base_ref }}
# PR base on pull_request; previous-tip on push. Empty for schedule.
BASE_SHA: ${{ github.event.pull_request.base.sha || github.event.before }}
# Real PR head commit (not the merge ref this job checks out), used to
# avoid rerunning e2e after a human commits only reviewed screenshots.
HEAD_SHA: ${{ github.event.pull_request.head.sha }}
# `update-screenshots` forces a full e2e run so every reference shot is
# re-rendered into the candidate artifact, even when the oracle would
# otherwise thin (subset) or skip the suite.
UPDATE_SCREENSHOTS_LABEL: ${{ contains(github.event.pull_request.labels.*.name, 'update-screenshots') }}
# `ci-full` forces the full e2e suite, bypassing the oracle's thinning
# entirely — for when a change wants exhaustive coverage NOW rather
# than waiting for the nightly net (e.g. infra/runner changes whose
# blast radius the selector-vocabulary map cannot see).
CI_FULL_LABEL: ${{ contains(github.event.pull_request.labels.*.name, 'ci-full') }}
run: |
if [ "$EVENT" = "schedule" ]; then
echo "Nightly schedule -> full e2e suite"
{ echo "mode=full"; echo "specs="; echo "unit_mode=full"; } >> "$GITHUB_OUTPUT"
exit 0
fi
# Screenshot-only head commit -> skip e2e. A human accepting a candidate
# artifact already reviewed the parent run's output, and the commit adds
# only reference PNGs, so re-running the suite would render the same shots
# again. Keep this before the label checks so a still-present request label
# does not turn the reviewed follow-up into another expensive full run.
if [ -n "$HEAD_SHA" ]; then
changed="$(git diff --name-only "${HEAD_SHA}^" "${HEAD_SHA}" 2>/dev/null || true)"
if [ -n "$changed" ] && ! printf '%s\n' "$changed" | grep -qv '^tests/e2e/screenshots/'; then
echo "HEAD commit only refreshes reference screenshots -> skip e2e"
{ echo "mode=skip"; echo "specs="; echo "unit_mode=full"; } >> "$GITHUB_OUTPUT"
exit 0
fi
fi
# A promotion is the release gate for the whole trunk batch. Always
# exercise the complete e2e suite here rather than allowing the oracle
# to thin or skip it. Keep this after the screenshot-only escape above:
# a reference-only follow-up already has immutable evidence from its
# parent and does not need another full render.
if [ "$EVENT" = "pull_request" ] && [ "$BASE_REF" = "release" ]; then
echo "PR targets release -> full e2e suite (promotion gate)"
{ echo "mode=full"; echo "specs="; echo "unit_mode=full"; } >> "$GITHUB_OUTPUT"
exit 0
fi
if [ "$UPDATE_SCREENSHOTS_LABEL" = "true" ]; then
echo "update-screenshots label -> full e2e suite (force screenshot refresh)"
{ echo "mode=full"; echo "specs="; echo "unit_mode=full"; } >> "$GITHUB_OUTPUT"
exit 0
fi
if [ "$CI_FULL_LABEL" = "true" ]; then
echo "ci-full label -> full e2e suite (oracle bypassed on request)"
{ echo "mode=full"; echo "specs="; echo "unit_mode=full"; } >> "$GITHUB_OUTPUT"
exit 0
fi
# No usable base (first push / forced push / branch create) -> full.
if [ -z "$BASE_SHA" ] || [ "$BASE_SHA" = "0000000000000000000000000000000000000000" ]; then
echo "No base sha for $EVENT -> full e2e suite"
{ echo "mode=full"; echo "specs="; echo "unit_mode=full"; } >> "$GITHUB_OUTPUT"
exit 0
fi
# full clone already has base history; this is belt-and-suspenders.
git fetch --no-tags --depth=1 origin "$BASE_SHA" 2>/dev/null || true
node scripts/test-oracle.mts --plan --base "$BASE_SHA" | tee plan.txt
grep -E '^(mode|specs|unit_mode|unit_specs)=' plan.txt >> "$GITHUB_OUTPUT"
{ echo '### e2e oracle plan'; echo '```'; cat plan.txt; echo '```'; } >> "$GITHUB_STEP_SUMMARY"
# Advisory: annotate which reference shots a UI change will refresh. Stale
# shots DON'T block the PR — the `e2e` job below already re-renders them as a
# side effect of the gate run, and `screenshot-artifacts` preserves the diff
# for explicit review. Pure static analysis — no build, Electron, or rerun.
# PR-only: no label exists for push/schedule.
- name: Plan reference-screenshot regeneration
if: github.event_name == 'pull_request'
env:
BASE_SHA: ${{ github.event.pull_request.base.sha }}
UPDATE_SCREENSHOTS_LABEL: ${{ contains(github.event.pull_request.labels.*.name, 'update-screenshots') }}
run: |
git fetch --no-tags --depth=1 origin "$BASE_SHA" 2>/dev/null || true
node scripts/check-screenshots.mts --plan --base "$BASE_SHA"
# Keep every full e2e run at 8 shards. This was sized when both runner
# classes had a tight memory ceiling: the private-repo hosted runner was
# 7 GB and each self-hosted `copse-e2e` container is capped at 6 GB. Public
# repos now get 4-vCPU / 16 GiB hosted runners, so on the default runner
# this fan-out is conservative — retune it against a run record rather
# than by assuming the headroom, and keep it safe for a fleet opt-in.
# Packing the suite into 3 shards makes each
# container launch too many sequential Electron sessions; accumulated
# Electron/gortex processes then OOM-crash unrelated late-running specs.
# `--shard` and the subset round-robin still partition specs disjointly.
- id: shards
env:
PLAN_MODE: ${{ steps.plan.outputs.mode }}
PLAN_SPECS: ${{ steps.plan.outputs.specs }}
run: |
# Size the matrix to the work the oracle actually planned. Previously
# this was always 8, so a screenshots-only commit (mode=skip) still
# dispatched 8 shards that each paid checkout + `npm ci` + dist
# download before hitting the early-exit in the run step — 8 self-hosted
# runners occupied to do nothing, on every bot screenshot push. Same for
# a subset smaller than 8: the surplus shards booted only to find an
# empty slice.
#
# An empty list must NOT reach the `e2e` matrix: GitHub Actions treats
# `strategy.matrix: []` as a job **failure**, not 'skipped' (observed on
# #1233 — `needs.e2e.result=failure` painted `CI Passed` red in skip
# mode). The `e2e` job `if:` therefore gates on `e2e_shard_total != 0`
# so a zero-shard plan skips the job cleanly; `ci-passed` then sees
# 'skipped' and its mode=skip branch accepts it.
if [ "$PLAN_MODE" = "skip" ]; then
echo "mode=skip -> no e2e shards dispatched"
{ echo "total=0"; echo "list=[]"; } >> "$GITHUB_OUTPUT"
exit 0
fi
if [ "$PLAN_MODE" = "subset" ]; then
# SHARD_TOTAL is the round-robin modulus in the run step, so total and
# the list must agree or specs would be assigned to shards that were
# never dispatched. Cap at 8 (the memory ceiling from #339).
n=0
for _ in $PLAN_SPECS; do n=$(( n + 1 )); done
[ "$n" -gt 8 ] && n=8
if [ "$n" -lt 1 ]; then
echo "mode=subset with no specs -> no e2e shards dispatched"
{ echo "total=0"; echo "list=[]"; } >> "$GITHUB_OUTPUT"
exit 0
fi
list="$(seq -s, 1 "$n")"
echo "mode=subset -> $n shard(s)"
{ echo "total=$n"; echo "list=[$list]"; } >> "$GITHUB_OUTPUT"
exit 0
fi
{ echo "total=8"; echo "list=[1,2,3,4,5,6,7,8]"; } >> "$GITHUB_OUTPUT"
# Auto-fix the PR branch: run every autofix we have — ESLint `--fix` then
# Prettier `--write` — and commit the result back onto the head branch. This
# is the "fixer" half of the format gate; `precheck` still runs `lint` +
# `format:check` as the "gate" half, which stays honest when this job is off
# (no PAT) or can't run (fork PR).
#
# Scope to files changed vs the PR base (not the whole tree): full-repo
# `eslint . --fix` regularly OOM-kills (exit 137) on the 6 GB check runners
# under fleet load, painting every PR red before precheck even starts —
# `prettier --write .` did too, before oxfmt replaced it. Diff-scoped autofix
# is enough — the gate half still checks the whole tree — and keeps peak
# memory proportional to the PR.
#
# A SEPARATE job rather than folded into `precheck`: precheck checks out the
# merge ref (fetch-depth: 0) so the e2e oracle can diff against base, whereas
# committing needs the branch (head_ref) checkout. The formatter runs last so it tidies
# whatever ESLint rewrote; `eslint-config-prettier` means the two never fight.
#
# Pushed as the copse-release-bot App rather than with the default
# GITHUB_TOKEN: GitHub won't start a run for a GITHUB_TOKEN push (recursion
# guard), so a token-pushed format commit would land as the PR head with no CI
# and leave the PR blocked. An App push is a distinct actor and fires
# `pull_request: synchronize` normally, and the top-level
# `concurrency: cancel-in-progress` supersedes THIS run with one on the
# formatted commit — so build/e2e aren't paid for twice. It terminates in one
# extra run: the re-triggered run reformats nothing (idempotent), so this job
# finds no diff and pushes nothing. Falls back to github.token when the App
# credentials are unset (commit lands, just no CI re-trigger — the behaviour
# before any push identity was configured).
autoformat:
if: github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name == github.repository
# Hosted by default; SELF_HOSTED_CHECKS=copse-checks opts this job onto the
# self-hosted check fleet (see the routing note above). Watch for 6 GB self-hosted OOM on eslint (see precheck).
runs-on: ${{ vars.SELF_HOSTED_CHECKS || 'ubuntu-latest' }}
timeout-minutes: 20
permissions:
contents: write
steps:
# Dispatched by `pull_request` but often EXECUTED minutes later, queued
# behind the fleet. A PR that merges in the meantime takes its head branch
# with it (auto-delete on merge), so checking out by branch name fails with
# "branch not found" and paints a red X on an already-merged PR. There is nothing to push
# formatting fixes to once the branch is gone; detect that first and no-op
# green. Skipping `changed` leaves `steps.changed.outputs.any` empty, which
# already gates setup and every step after it.
- name: Check head branch still exists
id: head
env:
GH_TOKEN: ${{ github.token }}
BRANCH: ${{ github.head_ref }}
# Self-hosted check-fleet images ship curl+jq but not the gh CLI
# (ci-runners/Dockerfile). Prefer the REST API over `gh api`.
GITHUB_API_URL: ${{ github.api_url }}
run: |
ENC_BRANCH=$(printf '%s' "$BRANCH" | jq -sRr @uri)
HTTP_STATUS=$(curl -sS -o /dev/null -w '%{http_code}' \
-H "Authorization: Bearer ${GH_TOKEN}" \
-H "Accept: application/vnd.github+json" \
-H "X-GitHub-Api-Version: 2022-11-28" \
"${GITHUB_API_URL}/repos/${{ github.repository }}/branches/${ENC_BRANCH}") || {
echo "::error::Could not query head branch '$BRANCH'; refusing to treat a transport failure as branch deletion."
exit 1
}
case "$HTTP_STATUS" in
200) echo "exists=true" >> "$GITHUB_OUTPUT" ;;
404)
echo "::notice::Head branch '$BRANCH' no longer exists (PR merged and branch deleted) — nothing to autoformat."
echo "exists=false" >> "$GITHUB_OUTPUT"
;;
*)
echo "::error::Head branch lookup returned HTTP $HTTP_STATUS; refusing to skip autoformat."
exit 1
;;
esac
# continue-on-error preserves that documented degradation: a missing or
# broken App credential leaves `outputs.token` empty and the checkout
# below falls through to github.token rather than reddening the PR.
- uses: actions/create-github-app-token@v3
id: app-token
if: steps.head.outputs.exists == 'true'
continue-on-error: true
with:
app-id: ${{ secrets.RELEASE_APP_ID }}
private-key: ${{ secrets.RELEASE_APP_PRIVATE_KEY }}
permission-contents: write
- uses: actions/checkout@v7.0.1
if: steps.head.outputs.exists == 'true'
with:
ref: ${{ github.head_ref }}
fetch-depth: 0
token: ${{ steps.app-token.outputs.token || github.token }}
# Decide whether there is anything to fix BEFORE installing. `npm ci` /
# cache-restore is ~2-5 min on this fleet and the autofix itself takes
# seconds, so on a diff with no formattable file the old order paid the
# whole install to print "nothing to do". Only git is needed to answer the
# question, and checkout has already run.
- id: changed
name: List autofixable changed files
if: steps.head.outputs.exists == 'true'
env:
BASE_SHA: ${{ github.event.pull_request.base.sha }}
run: |
if ! git cat-file -e "${BASE_SHA}^{commit}" 2>/dev/null; then
git fetch --no-tags --depth=1 origin "$BASE_SHA"
fi
# NUL-delimited via a temp file rather than a step output: a path is
# allowed to contain anything but NUL, and a step output cannot carry a
# newline safely.
git diff -z --name-only --diff-filter=ACMR "$BASE_SHA"...HEAD \
-- '*.cjs' '*.css' '*.cts' '*.js' '*.json' '*.jsx' '*.md' '*.mjs' \
'*.mts' '*.ts' '*.tsx' '*.yaml' '*.yml' \
> "${RUNNER_TEMP}/autofix-files.z" || true
if [ -s "${RUNNER_TEMP}/autofix-files.z" ]; then
echo "any=true" >> "$GITHUB_OUTPUT"
else
echo "No autofixable files changed vs base — skipping install and autofix."
echo "any=false" >> "$GITHUB_OUTPUT"
fi
- uses: ./.github/actions/setup
if: steps.changed.outputs.any == 'true'
# `|| true` on eslint: a non-autofixable lint error must not abort the run —
# we still want the fixes ESLint did apply to land. Real lint failures stay
# the job of precheck's `lint` gate, not this fixer. The formatter retries
# once on OOM (137); a second kill fails the job so a wedged runner is
# visible. oxfmt is a Rust binary and has not been seen to OOM here, but
# the retry costs nothing and this fleet is where the kills happened.
- name: Autofix changed files
if: steps.changed.outputs.any == 'true'
run: |
mapfile -d '' -t files < "${RUNNER_TEMP}/autofix-files.z"
echo "Autofixing ${#files[@]} changed file(s)"
npx eslint --fix -- "${files[@]}" || true
set +e
npx oxfmt --write -- "${files[@]}"
status=$?
set -e
if [ "$status" -eq 0 ]; then exit 0; fi
if [ "$status" -ne 137 ]; then exit "$status"; fi
echo "::warning::oxfmt OOM-killed (137); retrying once"
sleep 10
npx oxfmt --write -- "${files[@]}"
- name: Commit and push fixes
if: steps.changed.outputs.any == 'true'
run: |
if git diff --quiet; then
echo "No formatting changes to commit."
exit 0
fi
git config user.name "github-actions[bot]"
git config user.email "41898282+github-actions[bot]@users.noreply.github.com"
git commit -am "style: apply eslint --fix and oxfmt"
git push
check:
needs: precheck
if: github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name == github.repository
# Hosted by default; SELF_HOSTED_CHECKS=copse-checks opts this job onto the
# self-hosted check fleet (see the routing note above). History: coverage/esbuild has died with "service was stopped" on
# the 6 GB self-hosted fleet under load (same class as precheck OOM) — add
# headroom if it recurs.
runs-on: ${{ vars.SELF_HOSTED_CHECKS || 'ubuntu-latest' }}
timeout-minutes: 45
steps:
- uses: actions/checkout@v7.0.1
- uses: ./.github/actions/setup
# Preparation tests exercise the real kernel boundary and must not skip it.
- name: Install sandbox dependencies for unit integration tests
if: runner.environment == 'github-hosted' && runner.os == 'Linux'
run: |
set -euxo pipefail
sudo apt-get update
sudo apt-get install -y --no-install-recommends bubblewrap socat
if [[ -e /proc/sys/kernel/apparmor_restrict_unprivileged_userns ]]; then
sudo sysctl -w kernel.apparmor_restrict_unprivileged_userns=0
fi
bwrap --unshare-all --dev-bind / / --die-with-parent true
# Unit suite under c8 + coverage-baseline.json ratchet (replaces the
# separate coverage job — same tests, no duplicate npm test run).
#
# THINNING (stacked PRs only). A PR whose base is another PR's branch
# cannot merge yet: the layer below it has to land first, and when it does
# this PR is retargeted at trunk and runs this job again under the `full`
# arm. So the run that actually gates a merge into `main`/`release` is
# always the whole suite plus the coverage ratchet — thinning never buys a
# green that lets unrun tests reach trunk. That is the same "lowest
# unmerged PR gates the merge" rule the heavy tier already applies via
# `base_ref`, extended to the one tier that had no diff-awareness at all.
#
# Only `subset` skips the ratchet, and it must: a partial run produces a
# coverage number that is not comparable to the baseline, so gating on it
# would fail every thinned run. `skip` (docs-only) drops both.
# An unset `unit_mode` is read as `full`, so a plan branch that forgets to
# emit one fails safe.
- name: Unit tests
env:
# c8 retains V8 coverage until the suite exits. Test entries share
# split ESM chunks, so remapping scales with unique application code
# instead of hundreds of dependency-inlined standalone bundles.
#
# The peak is c8's post-run remap in a single process, after every
# test worker has already exited -- not the workers themselves, so
# TEST_FILE_CONCURRENCY does not bound it. 3072 aborted there with
# "Ineffective mark-compacts near heap limit" once the whole suite had
# passed; 6144 completes the remap and the ratchet. ubuntu-latest has
# 16 GB, and only this one process is live at that point.
NODE_OPTIONS: --max-old-space-size=6144
UNIT_MODE: ${{ needs.precheck.outputs.unit_mode }}
UNIT_SPECS: ${{ needs.precheck.outputs.unit_specs }}
# True only for a PR stacked on another PR's branch. `push`, tags and
# the nightly schedule have no base_ref and are never thinnable.
STACKED_PR: ${{ github.event_name == 'pull_request' && github.base_ref != 'main' && github.base_ref != 'release' }}
# Through the environment, never `${{ }}` inside the script: a base
# branch name is attacker-chosen text on any PR, and interpolating it
# into `run:` would splice it into this shell.
BASE_REF: ${{ github.base_ref }}
run: |
set -euo pipefail
if [ "$UNIT_MODE" = "skip" ]; then
echo "Oracle: docs-only diff — no unit test can observe it. Skipping the unit suite."
exit 0
fi
if [ "$STACKED_PR" = "true" ] && [ "$UNIT_MODE" = "subset" ] && [ -n "$UNIT_SPECS" ]; then
n=0
for _ in $UNIT_SPECS; do n=$(( n + 1 )); done
echo "Stacked PR (base '${BASE_REF}' is not trunk) -> $n selected unit file(s)."
echo "The trunk-targeted run of this PR will run the full suite + coverage ratchet."
# Deliberately unquoted: UNIT_SPECS is a space-separated file list.
# shellcheck disable=SC2086
npm test -- $UNIT_SPECS
exit 0
fi
npm run coverage:ci
# The console gets the `dot` reporter (see scripts/run-tests.mts) so a
# failure is reachable through the job-log API's ~5000-line tail. This is
# the full TAP for the same run, for when you want the passing detail too.
- uses: actions/upload-artifact@v7
if: always()
continue-on-error: true
with:
name: unit-tests-tap
path: unit-tests.tap
if-no-files-found: ignore
retention-days: 7
# Diagnostic HTML only — coverage:ci / coverage-gate already decided pass/fail.
# Org artifact-storage quota must not fail the merge gate after a green suite
# (seen on #1074: CreateArtifact quota error after 0 unit failures).
- uses: actions/upload-artifact@v7
if: always()
continue-on-error: true
with:
name: coverage-report
path: coverage/lcov-report
if-no-files-found: ignore
# Debug/inspection aid, not a durable record — expire quickly so it
# doesn't accumulate against the org's Actions storage quota.
retention-days: 7
# The CommonMark normalizer-parity check now lives in the extracted
# @copse/streaming-markdown repo's CI (copse-dev/streaming-markdown).
# The build is validated once in the dedicated `build` job below (whose
# dist artifact the e2e shards reuse), so it no longer runs here.
# Retrieval-quality gate for the semantic-search backend. This job indexes the
# repo and runs the fixture query set, asserting recall@k and search latency
# stay within the committed thresholds. A drop in ranking quality (or a gortex
# CLI/JSON change that breaks parsing) fails here rather than silently
# degrading in-app search.
bench:
needs: precheck
if: >-
(github.event_name != 'push' || github.ref != 'refs/heads/main') &&
(github.event_name != 'pull_request' ||
github.event.pull_request.head.repo.full_name == github.repository) &&
(github.event_name != 'pull_request' ||
github.base_ref == 'release' ||
contains(github.event.pull_request.labels.*.name, 'ci-full')) &&
(github.event_name != 'pull_request' ||
github.event.pull_request.draft == false ||
contains(github.event.pull_request.labels.*.name, 'ci-full'))
# HEAVY TIER. Runs on promotion PRs (base `release`), on pushes to `release`,
# on the nightly schedule and on release tags — not on the day-to-day PRs into
# `main`, and not on pushes to `main`. bench fetches gortex and runs
# the semantic-search + agent gates; paying that per PR when ~20 land a day
# is the cost the trunk/promotion model exists to amortise. The `ci-full`
# label forces it on a trunk-targeted PR that genuinely wants the signal.
# Draft PRs skip the heavy tier. Drafts cannot merge, so the expensive
# signal has no consumer until the PR is promoted — and `ready_for_review`
# (see the trigger block) dispatches the full tier the moment it is. The
# `ci-full` label forces it earlier for a draft that genuinely wants it.
# `precheck` / `check` / `autoformat` still run on drafts, so lint, types,
# unit tests and the autofixer keep giving fast feedback.
# Hosted by default; SELF_HOSTED_CHECKS=copse-checks opts this job onto the
# self-hosted check fleet (see the routing note above). No Electron/display need.
runs-on: ${{ vars.SELF_HOSTED_CHECKS || 'ubuntu-latest' }}
timeout-minutes: 45
steps:
- uses: actions/checkout@v7.0.1
- uses: ./.github/actions/setup
# Guarantee the gortex binary independently of the setup action. On a
# self-hosted runner whose baked image predates gortex, the baked-deps seed
# sets seeded=true (so npm ci + its postinstall fetch are skipped) yet the
# baked layer has no vendor/gortex to copy — leaving the binary absent.
# fetch-gortex is idempotent: a no-op when the (checksum-verified) binary is
# already present, a download when it isn't.
- run: node scripts/fetch-gortex.mts
# The floor is a breakage detector, not a target: a gortex CLI/JSON change
# that breaks parsing craters recall to ~0, and that's the main thing this
# catches. Recall is deterministic per corpus (~81% observed on 16 fixture
# queries), but unrelated PRs that add files shift rankings by a query or
# two, so the 0.65 floor leaves headroom to avoid flaking those builds. The
# p95 ceiling is generous vs the ~130ms observed locally (CI is slower, the
# daemon starts cold). Tighten either once the fixture set is larger.
- run: >-
node scripts/semantic-search-bench.mts --backend gortex --gate
--min-recall 0.65 --max-p95-ms 3000 --json bench-semantic.json
- uses: actions/upload-artifact@v7
if: always()
continue-on-error: true
with:
name: bench-semantic
path: bench-semantic.json
if-no-files-found: ignore
# Small trend JSON; keep two weeks for regression triage, then expire.
retention-days: 14
# Agent bench harness self-test (#752): the deterministic mock run drives
# run-agent-loop headlessly end-to-end (directive tool call → workspace
# write → grading) and the --gate ratchet compares solve rate and
# tokens-per-solve against benchmarks/bench-baseline.json. No model, ~30s.
- run: npm run bench:agent -- --mock --gate
# Doctrine ablation harness self-test (#744): runs the full and omit-tools
# arms through the real agent loop with a deterministic mock task. The
# real-model matrix stays in doctrine-eval-model below.
- run: >-
npm run eval:doctrine -- --provider mock --repeats 1 --sections tools
--out bench-results/doctrine-mock --require-solved --require-doctrine
# Steer A/B harness self-test: drives both arms of the mock pack through
# the real agent loop, including git workspace setup and the check runner.
# The mock ignores the system prompt, so this proves the harness works and
# says nothing about any steer — real-model lift is an on-demand run
# (docs/steer-evals.md), never a per-PR gate.
- run: >-
npm run eval:steer -- --provider mock --repeats 1
--out bench-results/steer-mock --require-gates
- uses: actions/upload-artifact@v7
if: always()
continue-on-error: true
with:
name: bench-agent-mock
path: bench-results/
if-no-files-found: ignore
# Small trend data; keep two weeks for regression triage, then expire.
retention-days: 14
# Real-model agent benchmark (#752): the SWE-bench Verified pinned subset
# driven by a local model on a self-hosted runner with an LM Studio host.
# Runs nightly (schedule) or on demand via the `bench-agent` PR label, and
# only when the LM_EVAL_RUNNER repo variable names a runner. Deliberately
# OUT of the ci-passed critical path (docs/testing-strategy.md): slow,
# non-deterministic trend data, not a merge gate.
bench-agent-model:
needs: precheck
if: >-
vars.LM_EVAL_RUNNER != '' &&
(github.event_name != 'pull_request' ||
github.event.pull_request.head.repo.full_name == github.repository) &&
(github.event_name == 'schedule' ||
contains(github.event.pull_request.labels.*.name, 'bench-agent'))
runs-on: ${{ vars.LM_EVAL_RUNNER }}
timeout-minutes: 90
steps:
- uses: actions/checkout@v7.0.1
- uses: ./.github/actions/setup
# Resolve the pinned instance ids into task files (network required).
- run: npm run bench:swe-tasks
- env:
LM_STUDIO_URL: ${{ vars.LM_STUDIO_URL }}
LM_STUDIO_MODEL: ${{ vars.LM_STUDIO_MODEL }}
LM_STUDIO_API_KEY: ${{ secrets.LM_STUDIO_API_KEY }}
run: npm run bench:agent -- --tasks benchmarks/tasks/swe-bench --gate
- uses: actions/upload-artifact@v7
if: always()
continue-on-error: true
with:
name: bench-agent-model
path: bench-results/
if-no-files-found: ignore
# Small trend data; keep two weeks for regression triage, then expire.
retention-days: 14
# Working-style doctrine prompt-section A/B (#744): same fixed task subset,
# model, and sampling posture for full vs omitted prompt sections. Runs only
# nightly or when a PR is explicitly labelled `bench-doctrine`, and only on a
# configured LM Studio eval runner. Trend report only; never a merge gate.
doctrine-eval-model:
needs: precheck
if: >-
vars.LM_EVAL_RUNNER != '' &&
(github.event_name != 'pull_request' ||
github.event.pull_request.head.repo.full_name == github.repository) &&
(github.event_name == 'schedule' ||
contains(github.event.pull_request.labels.*.name, 'bench-doctrine'))
runs-on: ${{ vars.LM_EVAL_RUNNER }}
timeout-minutes: 90
steps:
- uses: actions/checkout@v7.0.1
- uses: ./.github/actions/setup
- env:
LM_STUDIO_URL: ${{ vars.LM_STUDIO_URL }}
LM_STUDIO_MODEL: ${{ vars.LM_STUDIO_MODEL }}
LM_STUDIO_API_KEY: ${{ secrets.LM_STUDIO_API_KEY }}
run: >-
npm run eval:doctrine -- --provider lmstudio --repeats 3
--sections tools,workingStyle,gitBranchSafety
--out bench-results/doctrine
- uses: actions/upload-artifact@v7
if: always()
continue-on-error: true
with:
name: doctrine-eval-model
path: bench-results/doctrine/
if-no-files-found: ignore
retention-days: 30
# Tool-preference eval (#1845): asked a read-only "what landed on main, is CI
# green?" question, does the agent answer through the first-class git/gh/CI
# tools, or does it drive `gh` and network `git` through run_shell and charge
# the user an external-shell approval for each? The exported thread in that
# issue took five. A GA blocker, but deliberately NOT a merge gate: the
# scoring it rests on is unit-tested on every PR
# (scripts/lib/eval-scenario-git-ci-tools.test.ts, which replays that thread),
# while this lane measures what real models actually do. Nightly, or on demand
# via the `eval-tool-preference` PR label, and only on a configured eval
# runner.
#
# The matrix is what the issue's acceptance criteria ask for: the native loop
# across the prompt class's wordings, plus one ACP path, whose namespaced
# bridged calls the scorer matches through `matchesBridgedToolName`.
# `fail-fast` is off because which wording an agent trips on IS the finding —
# stopping the matrix at the first red would throw that away.
eval-tool-preference:
needs: precheck
if: >-
vars.LM_EVAL_RUNNER != '' &&
(github.event_name != 'pull_request' ||
github.event.pull_request.head.repo.full_name == github.repository) &&
(github.event_name == 'schedule' ||
contains(github.event.pull_request.labels.*.name, 'eval-tool-preference'))
runs-on: ${{ vars.LM_EVAL_RUNNER }}
timeout-minutes: 60
strategy:
fail-fast: false
matrix:
include:
- { path: native, model: '', variant: '0' }
- { path: native, model: '', variant: '1' }
- { path: native, model: '', variant: '2' }
# The ACP arm runs the wording taken verbatim from the exported
# thread, so a regression there is comparable to the recorded case.
- { path: acp-codex, model: 'acp:codex-acp', variant: '0' }
steps:
- uses: actions/checkout@v7.0.1
- uses: ./.github/actions/setup
# The eval drives the packaged renderer through Electron, not the dev
# server, so the bundle has to exist before WDIO launches.
- run: npm run build
- env:
COPSE_EVAL_SCENARIO: tests/e2e/scenarios/git-ci-first-class-tools.json
COPSE_EVAL_PROMPT_VARIANT: ${{ matrix.variant }}
COPSE_EVAL_MODEL: ${{ matrix.model }}
LM_STUDIO_URL: ${{ vars.LM_STUDIO_URL }}
LM_STUDIO_MODEL: ${{ vars.LM_STUDIO_MODEL }}
LM_STUDIO_API_KEY: ${{ secrets.LM_STUDIO_API_KEY }}
run: npm run test:e2e:agent-eval
# Publish the pass/fail artifact even when the step above already failed
# its in-run asserts: a run that violated the expectations is exactly the
# one whose scored trace is worth reading. Same scoring a human gets from
# `npm run analyze:thread` on the captured JSONL, so the lane and a local
# investigation cannot disagree.
- if: always()
shell: bash
run: |
artifact=$(ls -t tests/e2e/artifacts/git-ci-first-class-tools-*.jsonl 2>/dev/null | head -1)
if [ -z "$artifact" ]; then
echo "no eval artifact captured — the run died before writing a thread"
exit 1
fi
report="tests/e2e/artifacts/report-${{ matrix.path }}-${{ matrix.variant }}.json"
npm run --silent analyze:thread -- "$artifact" \
tests/e2e/scenarios/git-ci-first-class-tools.json > "$report"
cat "$report"
- uses: actions/upload-artifact@v7
if: always()
continue-on-error: true
with:
name: eval-tool-preference-${{ matrix.path }}-${{ matrix.variant }}
path: tests/e2e/artifacts/
if-no-files-found: ignore
# Trend + triage data for a GA gate; a month covers a release cycle.
retention-days: 30
# Scratch-path GA bar (#1846): a sandboxed agent asked to stage intermediate
# output somewhere temporary must use the $TMPDIR the sandbox hands it (or the
# workspace), never a hardcoded /tmp/... — which the scope heuristic classes as
# external and charges the user an approval for. Deterministic scoring lives in
# `npm test` (eval-scratch-paths / eval-tool-expectations); this job is the
# behavioural half, so it needs a real model and the real Electron app.
#
# Runs nightly or on the `bench-agent-eval` label, on the same LM Studio eval
# runner as doctrine-eval-model. Every prompt variant runs even after one
# fails: the wording is meant to vary while the intent stays fixed, so one red
# phrasing out of three is a different finding from three reds, and fail-fast
# would report them as the same thing.
scratch-path-eval-model:
needs: precheck
if: >-
vars.LM_EVAL_RUNNER != '' &&
(github.event_name != 'pull_request' ||
github.event.pull_request.head.repo.full_name == github.repository) &&
(github.event_name == 'schedule' ||
contains(github.event.pull_request.labels.*.name, 'bench-agent-eval'))
runs-on: ${{ vars.LM_EVAL_RUNNER }}
timeout-minutes: 120
steps:
- uses: actions/checkout@v7.0.1
- uses: ./.github/actions/setup
- run: npm run build
# The ACP arm only runs where an adapter is actually installed, so the
# agent is named by a repo variable (e.g. `acp:codex-acp`) rather than
# assumed. Unset, the native arm still gates on its own.
- name: Agent scratch-path eval
env:
COPSE_EVAL_SCENARIO: tests/e2e/scenarios/tmpdir-scratch-eval.json
COPSE_ACP_EVAL_MODEL: ${{ vars.ACP_EVAL_MODEL }}
LM_STUDIO_URL: ${{ vars.LM_STUDIO_URL }}
COPSE_EVAL_LOCAL_SERVER_URL: ${{ vars.LM_STUDIO_URL }}
LM_STUDIO_API_KEY: ${{ secrets.LM_STUDIO_API_KEY }}
run: |
set -uo pipefail
failed=0
for arm in native acp; do
if [ "$arm" = acp ]; then
[ -n "${COPSE_ACP_EVAL_MODEL:-}" ] || { echo "::notice::ACP_EVAL_MODEL unset, skipping the ACP arm"; continue; }
export COPSE_EVAL_MODEL="$COPSE_ACP_EVAL_MODEL"
else
unset COPSE_EVAL_MODEL
fi
for variant in 0 1 2; do
echo "::group::$arm prompt variant $variant"
COPSE_EVAL_PROMPT_VARIANT="$variant" npm run test:e2e:agent-eval || failed=1
echo "::endgroup::"
done
done
exit "$failed"
- uses: actions/upload-artifact@v7
if: always()
continue-on-error: true
with:
name: scratch-path-eval-model
# The JSONL thread export per run. A failure message names the
# offending command; the trace is how you see what led to it.
path: tests/e2e/artifacts/
if-no-files-found: ignore
retention-days: 30
build:
needs: precheck
if: >-
(github.event_name != 'pull_request' ||
github.event.pull_request.head.repo.full_name == github.repository) &&
(github.event_name != 'pull_request' ||
github.event.pull_request.draft == false ||
contains(github.event.pull_request.labels.*.name, 'ci-full'))
# Draft PRs skip the heavy tier. Drafts cannot merge, so the expensive
# signal has no consumer until the PR is promoted — and `ready_for_review`
# (see the trigger block) dispatches the full tier the moment it is. The
# `ci-full` label forces it earlier for a draft that genuinely wants it.
# `precheck` / `check` / `autoformat` still run on drafts, so lint, types,
# unit tests and the autofixer keep giving fast feedback.
# Hosted by default; SELF_HOSTED_CHECKS=copse-checks opts this job onto the
# self-hosted check fleet (see the routing note above). Produces the dist artifact consumed by the e2e shards below.
runs-on: ${{ vars.SELF_HOSTED_CHECKS || 'ubuntu-latest' }}
timeout-minutes: 30
# Build dist once and hand it to every e2e shard. Previously each shard ran
# its own `npm run build`; an earlier attempt to share dist via upload/
# download dropped some Monaco worker files, breaking specs that open the
# diff editor. Packing dist into a single tar before upload preserves the
# full tree (worker files, nested dirs, exec bits) verbatim, so the shards
# get a byte-identical dist without each rebuilding it.
steps:
- uses: actions/checkout@v7.0.1
- uses: ./.github/actions/setup
- run: npm run build
# Plain-Chrome geometry coverage replaces Electron specs that do not need
# main/preload IPC. Skip screenshot-only heads: their parent already ran
# this tier and attached its immutable render evidence.
- run: npm run build:demo
if: needs.precheck.outputs.mode != 'skip'
- run: npm run test:demo
if: needs.precheck.outputs.mode != 'skip'
- name: Collect changed browser reference screenshots
if: >-
success() && github.event_name == 'pull_request' &&
needs.precheck.outputs.mode != 'skip'
run: |
mkdir -p changed/tests/e2e/screenshots
touch changed/.demo
mapfile -t files < <(
{
git diff --name-only -- tests/e2e/screenshots/
git ls-files --others --exclude-standard -- tests/e2e/screenshots/
} | sort -u
)
for f in "${files[@]}"; do
[ -n "$f" ] || continue
cp "$f" "changed/tests/e2e/screenshots/"
done
echo "Collected ${#files[@]} browser screenshot(s)"
- uses: actions/upload-artifact@v7
if: >-
success() && github.event_name == 'pull_request' &&
needs.precheck.outputs.mode != 'skip'
with:
name: screenshots-demo
path: changed/
include-hidden-files: true
if-no-files-found: error
- uses: actions/upload-artifact@v7
if: failure()
with:
name: demo-failure-artifacts
path: e2e-failure-artifacts/
if-no-files-found: ignore
retention-days: 3
- run: tar -cf dist.tar dist
- uses: actions/upload-artifact@v7
with:
name: dist-tar
path: dist.tar
if-no-files-found: error
retention-days: 1
e2e:
needs: [build, precheck]
# The oracle decides the cost, not whether uncertain changes receive a gate:
# `subset` runs only the specs it can map confidently; `full` means the map is
# broad or LOW-confidence, so every shard runs. Treating `full` as a reason to
# skip e2e inverted the risk policy — the least-understood changes could merge
# with less integration coverage than confidently mapped UI changes.
#
# Draft PRs are unaffected: the `draft == false` clause below still applies,
# so neither a subset nor a full plan dispatches shards on work in progress.
if: >-
needs.precheck.outputs.e2e_shard_total != '0' &&
(github.event_name != 'push' || github.ref != 'refs/heads/main') &&
(github.event_name != 'pull_request' ||
github.event.pull_request.head.repo.full_name == github.repository) &&
(github.event_name != 'pull_request' ||
github.base_ref == 'release' ||
needs.precheck.outputs.mode == 'full' ||
needs.precheck.outputs.mode == 'subset' ||
contains(github.event.pull_request.labels.*.name, 'ci-full') ||
contains(github.event.pull_request.labels.*.name, 'update-screenshots')) &&
(github.event_name != 'pull_request' ||
github.event.pull_request.draft == false ||
contains(github.event.pull_request.labels.*.name, 'ci-full'))
# Zero-shard plans (oracle mode=skip / empty subset) skip this job via
# `e2e_shard_total != 0` — an empty matrix is a GHA **failure**, not a skip
# (see the precheck shards step).
# Full mode is intentionally expensive: it is the fail-safe result for a
# change the selector map cannot bound. Confident subsets keep ordinary UI
# changes cheap; docs-only changes produce zero shards and skip this job.
# `ci-full` remains the explicit override for a draft or a change whose
# author wants exhaustive coverage despite a subset plan.
#
# `update-screenshots` also runs this job on a trunk PR: screenshot
# candidates are produced by the e2e run, so without it the label would be
# inert on a plan that plans no shards.
# Draft PRs skip e2e for the same reason as `build` above; `build` is also
# gated, so on a draft this job's `needs` are unsatisfied and it would skip
# regardless — the explicit clause keeps the intent readable at this job.
# e2e runs on GitHub-hosted ubuntu-latest by default, for every event.
# The self-hosted Docker fleet (see ci-runners/) is opt-in via
# SELF_HOSTED_E2E=copse-e2e. The fleet was the default while this repo was
# private, when hosted meant 2-core/7GB — the box most of the shard
# quarantines exist to work around. A public repo gets 4-vCPU / 16 GiB
# hosted runners for free, which is more headroom than the 6 GB `copse-e2e`
# container, so the default no longer trades capacity for cost.
#
# Inverting this also removes a wedge: the old expression treated any
# SELF_HOSTED_E2E value other than the exact string 'ubuntu-latest' —
# including UNSET — as "use the fleet". With no registered `copse-e2e`
# runner, deleting one variable queued every PR/push e2e job indefinitely.
# Now the unset case is hosted, so the failure mode of a missing or
# mistyped variable is "runs on hosted", not "never runs".
#
# Runner is still picked by TRUST and still fails closed:
# - Nightly `schedule` and trusted push/same-repo PRs default to hosted.
# - The opt-in label set requires BOTH `self-hosted` AND `copse-e2e`, so an
# opted-in job lands only on the Linux Docker runners, never the macOS
# `self-hosted` boxes (which lack `copse-e2e`).
# - Fork PRs are skipped by the `if` guard above and must never run on any
# runner. There is deliberately no fork branch here: if that guard were
# ever removed, this expression evaluates to a falsy value (not a runner
# label) so the job fails closed instead of dispatching untrusted code
# anywhere — including onto a free hosted runner.
runs-on: >-
${{ github.event_name == 'schedule' && fromJSON('["ubuntu-latest"]') ||
((github.event_name == 'push' ||
(github.event_name == 'pull_request' &&
github.event.pull_request.head.repo.full_name == github.repository))
&& (vars.SELF_HOSTED_E2E == 'copse-e2e'
&& fromJSON('["self-hosted", "copse-e2e"]')
|| fromJSON('["ubuntu-latest"]'))) }}
timeout-minutes: 45
# Split the seeded e2e suite (see wdio.ci.conf.ts) into parallel shards so the
# whole gate finishes well inside the runner window. Each shard runs a disjoint
# slice of specs via wdio's `--shard current/total`. The shared `dist` artifact
# from the `build` job is unpacked here rather than rebuilt per shard.
#
# Every full run uses 8 shards. Each shard runs its slice as one sequential
# wdio process, relaunching Electron per spec. Both the hosted runner (7 GB)
# and self-hosted container (6 GB cap) eventually exhaust memory when a shard
# launches too many Electron/gortex processes; a later unrelated spec then
# fails to boot or OOM-crashes ("shutdown signal"). #339's data showed that
# 8 shards stays under that limit even in the densest spec range.
# The per-attempt `timeout` is 480s (step cap 18 min for 2 attempts) so attempt
# 1 finishes instead of being SIGKILL'd mid-retry, letting attempt 2 relaunch.
strategy:
# Fail fast: the first shard to go red cancels the rest. On the self-hosted
# fleet a failure is usually a whole-run problem (a poisoned dep cache, a
# broken commit, a runner-env issue) that every shard would hit anyway, so
# cancelling the siblings frees the box instead of grinding them all through
# their retry budget. Tradeoff: you see the first failing shard, not every
# failing shard, in one run.
#
# `ci-full` is the exception, because there the tradeoff inverts. That label
# already means "run the whole heavy tier on a trunk PR" — it is opt-in and
# rare, and it is used precisely when the e2e signal is what's wanted. With
# independent reds (not a whole-run problem) fail-fast surfaces exactly one
# per run: #1425 spent five runs learning about six specs, cancelling six
# shards each time, because each red hid the next. Letting a `ci-full` run
# report every shard costs fleet time on a PR that is already failing, and
# buys the complete list in one pass.
#
# Non-PR events have no labels, so `contains()` is false and fail-fast stays
# on for pushes, the nightly schedule, and tags — the fleet-protecting
# default is unchanged everywhere it was doing its job.
fail-fast: ${{ !contains(github.event.pull_request.labels.*.name, 'ci-full') }}
matrix:
shard: ${{ fromJSON(needs.precheck.outputs.e2e_shards) }}
env:
SHARD_TOTAL: ${{ needs.precheck.outputs.e2e_shard_total }}
steps:
- uses: actions/checkout@v7.0.1
- uses: ./.github/actions/setup
# Reclaim ~20-30 GB of runner-preloaded toolchains (dotnet/android/ghc/
# CodeQL/swift) and cached Docker images. The e2e shards have hit "No space
# left on device" mid-run, which cascades into OOM / runner-shutdown and
# Electron session-startup failures; freeing space up front removes that
# whole failure class. Best-effort — never fail the job on cleanup. Only on
# GitHub-hosted runners (the nightly) — never rm system dirs on a
# self-hosted box.
- name: Free disk space
if: runner.environment == 'github-hosted'
run: |
sudo rm -rf /usr/share/dotnet /usr/local/lib/android /opt/ghc \
/opt/hostedtoolcache/CodeQL /usr/share/swift /usr/local/share/boost || true
sudo docker image prune --all --force >/dev/null 2>&1 || true
df -h /
# Outer retry: actions/download-artifact marks intermittent CDN 403s
# ("Error from intermediary") as non-retryable, and with fail-fast:true a
# single shard's download flake cancels the whole e2e matrix (see
# actions/download-artifact#464 and main run 29591165512).
# GitHub-hosted images do not ship ASRT's Linux backend, and hosted is now
# the default for every e2e event (nightly `schedule` included) — the
# fleet is opt-in via SELF_HOSTED_E2E. Executable plugins fail closed
# without bubblewrap+socat (selected-plugin-browser on tip 528f92ce /
# runs 31781384161 and 31870672634), so the hosted-only step below
# installs them. Self-hosted runners bake both into the image already
# (ci-runners/Dockerfile), which is why it stays gated on
# `runner.environment`.
- name: Install Linux sandbox dependencies
if: runner.environment == 'github-hosted'
run: |
set -euxo pipefail
sudo apt-get update
sudo apt-get install -y --no-install-recommends bubblewrap socat
# Ubuntu 24.04's AppArmor userns restriction lets bwrap create a
# namespace but strips loopback setup (Failed RTM_NEWADDR). The
# self-hosted host bootstrap clears the same sysctl; do that here
# when the hosted runner permits it so ASRT can actually start.
if [[ -e /proc/sys/kernel/apparmor_restrict_unprivileged_userns ]]; then
sudo sysctl -w kernel.apparmor_restrict_unprivileged_userns=0
fi
command -v bwrap
command -v socat
bwrap --unshare-all --dev-bind / / --die-with-parent true
# The install above is hosted-only because the self-hosted image bakes
# both in (ci-runners/Dockerfile). When a self-hosted runner comes up on a
# stale image that assumption breaks silently — nothing installs, nothing
# checks, and the first sign is a spec failing ~40 minutes later with a
# product-shaped message: `selected-plugin-browser` reporting "Executable
# plugin behavior requires Copse's active OS sandbox", or
# `capability-lease-approval` waiting on a sandboxed turn-tree that never
# renders. Both read as regressions in whichever PR drew the runner, which
# is the expensive part. Run the same three checks the install step
# already ends with, on every runner, and fail here — where the message
# names the cause.
- name: Verify sandbox preconditions
if: >-
needs.precheck.outputs.mode == 'full' ||
needs.precheck.outputs.mode == 'subset'
run: |
set -uo pipefail
missing=''
command -v bwrap >/dev/null 2>&1 || missing="${missing} bubblewrap(bwrap)"
command -v socat >/dev/null 2>&1 || missing="${missing} socat"
if [ -n "${missing}" ]; then
probe='skipped (tools missing)'
elif bwrap --unshare-all --dev-bind / / --die-with-parent true >/dev/null 2>&1; then
probe='ok'
else
probe='FAILED'
fi
if [ -n "${missing}" ] || [ "${probe}" = 'FAILED' ]; then
echo "runner=${RUNNER_NAME:-unknown} environment=${RUNNER_ENVIRONMENT:-unknown}"
echo "missing:${missing:- none}"
echo "bwrap probe: ${probe}"
echo "::error title=e2e runner is not provisioned for the OS sandbox::missing:${missing:- none}; bwrap probe=${probe}. Specs that need the sandbox fail closed with product-shaped messages, so this fails here instead. Self-hosted runners get these from ci-runners/Dockerfile — this host is most likely on a stale image."
exit 1
fi
echo "sandbox preconditions ok (bwrap + socat present, probe passed)"
- uses: ./.github/actions/download-artifact-retry
with:
name: dist-tar
- run: tar -xf dist.tar
# The Electron/Chromedriver e2e session occasionally fails to start on the
# runner ("DevToolsActivePort file doesn't exist" / "POST /session" timeout).
# The hang wedges the whole wdio run, so in-process retries don't recover —
# only a fresh xvfb+Electron launch does. Retry the entire shard, cleaning
# orphaned session processes before each attempt and when the step exits.
# The runners are one job per PID-isolated container, so this cannot touch
# another job. Wrap each attempt in `timeout` so a plain hang cannot stall
# the job indefinitely.
# Oracle gate (see the precheck job's plan step): `full` shards the whole
# suite; `subset` distributes only the planned specs round-robin across the
# same 8 shards (so no shard exceeds the per-shard memory budget the count
# is tuned for); `skip` runs nothing. Every shard still reports a status, so
# this composes with any required check without depending on an aggregate
# gate first.
- name: e2e shard ${{ matrix.shard }}/${{ env.SHARD_TOTAL }} (oracle-gated, retry on flaky session startup)
if: >-
needs.precheck.outputs.mode == 'full' ||
needs.precheck.outputs.mode == 'subset'
timeout-minutes: 18
env:
PLAN_MODE: ${{ needs.precheck.outputs.mode }}
PLAN_SPECS: ${{ needs.precheck.outputs.specs }}
run: |
if [ "$PLAN_MODE" = "subset" ]; then
# Round-robin the planned specs onto this shard; an empty slice means
# this shard has nothing to do (subset smaller than 8) — report green.
i=0
SPEC_ARGS=""
for s in $PLAN_SPECS; do
if [ $(( i % SHARD_TOTAL )) -eq $(( ${{ matrix.shard }} - 1 )) ]; then
SPEC_ARGS="$SPEC_ARGS --spec $s"
fi
i=$(( i + 1 ))
done
if [ -z "$SPEC_ARGS" ]; then
echo "shard ${{ matrix.shard }}: no specs in this subset slice — nothing to run"
exit 0
fi
SHARD_ARGS=""
echo "oracle subset shard ${{ matrix.shard }}:${SPEC_ARGS}"
else
SHARD_ARGS="--shard ${{ matrix.shard }}/${{ env.SHARD_TOTAL }}"
SPEC_ARGS=""
fi
# `timeout` is GNU coreutils — present on Linux, but on macOS it's
# `gtimeout` (from `brew install coreutils`). Pick whichever exists;
# fall back to no inner timeout (the step `timeout-minutes` still caps
# a wedged run) so the command can't die with "timeout: not found".
TIMEOUT="$(command -v timeout || command -v gtimeout || true)"
# ASRT's Linux network bridge, matched by the socket path it owns.
#
# `@anthropic-ai/sandbox-runtime` spawns socat on both sides of the
# sandbox boundary, and both forms name the same generated socket
# (`linux-sandbox-utils.js`, initializeLinuxNetworkBridge):
#
# host: socat UNIX-LISTEN:$TMPDIR/claude-http-<hex>.sock,… TCP:localhost:<port>,…
# sandbox: socat TCP-LISTEN:3128,… UNIX-CONNECT:$TMPDIR/claude-http-<hex>.sock
#
# Matching that socket name — rather than the bare word `socat` — is
# what makes this safe to run on a shared runner. A blanket
# `pkill -f socat` would also reap a socat the runner infrastructure
# itself depends on, which is why #1561 recorded the leak but left
# the fix alone. Nothing but ASRT creates `claude-{http,socks}-*.sock`.
E2E_SOCAT_PATTERN='[s]ocat.*claude-(http|socks)-[0-9a-f]+\.sock'
cleanup_e2e_processes() {
# Bracketed patterns avoid matching this shell's own command line.
# A failed deleteSession can orphan any of these; self-hosted runner
# containers are reused, so clean before attempt 1 as well as retries.
pkill -TERM -f '[e]lectron/dist/electron' 2>/dev/null || true
pkill -TERM -f '[e]lectron-chromedriver/bin/chromedriver' 2>/dev/null || true
pkill -TERM -f '[X]vfb' 2>/dev/null || true
pkill -TERM -f '[g]ortex' 2>/dev/null || true
# Added for #1442's socat bridges: the reaper predates them, so they
# survived into the next job on a reused container. Each leaked
# bridge holds its listening socket, and a shard that then cannot
# start a session fails every spec that calls `reloadSession`.
pkill -TERM -f "$E2E_SOCAT_PATTERN" 2>/dev/null || true
sleep 1
pkill -KILL -f '[e]lectron/dist/electron' 2>/dev/null || true
pkill -KILL -f '[e]lectron-chromedriver/bin/chromedriver' 2>/dev/null || true
pkill -KILL -f '[X]vfb' 2>/dev/null || true
pkill -KILL -f '[g]ortex' 2>/dev/null || true
pkill -KILL -f "$E2E_SOCAT_PATTERN" 2>/dev/null || true
# The sockets outlive the processes that listened on them. Leaving
# them behind fills $TMPDIR on a long-lived runner and makes the
# diagnostics harder to read.
find "${TMPDIR:-/tmp}" -maxdepth 1 \
\( -name 'claude-http-*.sock' -o -name 'claude-socks-*.sock' \) \
-delete 2>/dev/null || true
}
capture_e2e_runner_diagnostics() {
local attempt="$1"
local attempt_status="$2"
local diagnostics_dir="e2e-failure-artifacts"
local diagnostics_file="${diagnostics_dir}/runner-attempt-${attempt}.txt"
mkdir -p "$diagnostics_dir"
{
echo "captured_at=$(date --iso-8601=seconds 2>/dev/null || date)"
echo "attempt=$attempt"
echo "exit_status=$attempt_status"
echo "runner_name=${RUNNER_NAME:-unknown}"
echo "runner_os=${RUNNER_OS:-unknown}"
echo "runner_arch=${RUNNER_ARCH:-unknown}"
echo "runner_environment=${RUNNER_ENVIRONMENT:-unknown}"
echo "github_run_id=${GITHUB_RUN_ID:-unknown}"
echo "github_job=${GITHUB_JOB:-unknown}"
echo "shard=${{ matrix.shard }}/${SHARD_TOTAL}"
echo
echo "## system"
uname -a || true
cat /etc/hostname 2>/dev/null || true
echo
echo "## cgroup"
# Docker uses cgroup v2 on the burst hosts, while the v1 files
# keep the artifact useful on older/self-managed Linux runners.
for cgroup_file in \
/sys/fs/cgroup/memory.current \
/sys/fs/cgroup/memory.peak \
/sys/fs/cgroup/memory.max \
/sys/fs/cgroup/memory.events \
/sys/fs/cgroup/memory.events.local \
/sys/fs/cgroup/pids.current \
/sys/fs/cgroup/pids.peak \
/sys/fs/cgroup/pids.max \
/sys/fs/cgroup/cpu.stat \
/sys/fs/cgroup/io.stat \
/sys/fs/cgroup/memory/memory.usage_in_bytes \
/sys/fs/cgroup/memory/memory.max_usage_in_bytes \
/sys/fs/cgroup/memory/memory.limit_in_bytes \
/sys/fs/cgroup/memory/memory.failcnt \
/sys/fs/cgroup/pids/pids.current \
/sys/fs/cgroup/pids/pids.max; do
if [ -r "$cgroup_file" ]; then
echo "### $cgroup_file"
cat "$cgroup_file"
fi
done
echo
echo "## memory"
free -h || true
echo
echo "## processes by RSS"
ps -eo pid,ppid,state,etimes,rss,vsz,comm --sort=-rss | head -n 80 || true
echo
# An RSS-sorted top-80 hides light, numerous processes — which is
# exactly the shape a leak takes. socat in particular is tiny, so a
# runner carrying dozens of orphans looks unremarkable above.
echo "## process counts by name"
ps -eo comm= | sort | uniq -c | sort -rn | head -n 30 || true
echo
echo "## e2e / sandbox process detail (orphans have ppid 1)"
ps -eo pid,ppid,etimes,comm,args \
| grep -E '[s]ocat|[b]wrap|[e]lectron|[c]hromedriver|[X]vfb' \
| head -n 60 || true
echo
# ASRT needs bubblewrap AND socat, and bwrap additionally needs
# usable unprivileged user namespaces at runtime (see
# ci-runners/docker-compose.yml). When these are wrong the app does
# not crash — it degrades to sandbox-off — so the fault is invisible
# unless captured here.
echo "## sandbox preconditions"
echo "socat=$(command -v socat || echo MISSING)"
echo "bwrap=$(command -v bwrap || echo MISSING)"
bwrap --version 2>&1 || true
for sysctl_file in \
/proc/sys/kernel/apparmor_restrict_unprivileged_userns \
/proc/sys/kernel/unprivileged_userns_clone \
/proc/sys/user/max_user_namespaces; do
if [ -r "$sysctl_file" ]; then
echo "$sysctl_file=$(cat "$sysctl_file")"
else
echo "$sysctl_file=UNREADABLE"
fi
done
# The same shape ASRT asks for. Proves the chain rather than just
# the binary's presence.
echo "### bwrap probe"
bwrap --unshare-all --dev-bind / / --die-with-parent true 2>&1 \
&& echo "bwrap_probe=ok" || echo "bwrap_probe=FAILED"
echo
echo "## open file descriptors"
echo "fd_nr=$(cat /proc/sys/fs/file-nr 2>/dev/null || echo unknown)"
echo
echo "## filesystems"
df -h / /tmp /dev/shm || true
echo
echo "## limits"
ulimit -a || true
} >"$diagnostics_file" 2>&1
echo "Saved runner diagnostics to $diagnostics_file"
# Also print it. The artifact is the durable copy, but downloading a
# workflow artifact needs a token with `actions:read`, which a GitHub
# App integration does not get — so for anyone triaging through the
# API the file may as well not exist (`403 Resource not accessible by
# integration`). The job log has no such gate. Collapsed by default,
# so a green-eyed scroll past a failing shard costs nothing.
echo "::group::runner diagnostics (attempt $attempt)"
cat "$diagnostics_file" || true
echo "::endgroup::"
}
# Dump whatever WebdriverIO and chromedriver wrote about the session.
# "Unable to connect to http://localhost:PORT" tells us the driver
# stopped accepting — it never says why, because the only account of
# that is the driver's own log, which lives in wdio's outputDir and was
# being discarded at job end.
#
# Filtered, not tailed. The job-log API returns roughly the last 5000
# lines, so a raw tail of ~20 driver logs (more with chromedriver's
# `verbose`) crowds out everything printed after it — which is how both
# the spec summary and the runner diagnostics became unreachable
# through the API even though they were right there in the file. Print
# the error-adjacent lines instead, and fall back to a raw tail only
# when nothing matched, so a novel failure is never silently dropped.
# The uploaded artifact remains the complete record.
E2E_LOG_SIGNALS='ERROR|WARN|Unable to connect|ECONNREFUSED|ECONNRESET|EPIPE|UND_ERR|bind\(\) failed|Cannot start http server|DevTools HTTP Request failed|aborted due to timeout|Timed out receiving message|no such session|invalid session id|session not created|boot-complete|was started successfully'
dump_e2e_session_logs() {
local logs_dir="e2e-failure-artifacts/wdio-logs"
[ -d "$logs_dir" ] || return 0
echo "::group::wdio + chromedriver logs (attempt $1)"
for log_file in "$logs_dir"/*.log; do
[ -f "$log_file" ] || continue
echo "### $log_file"
# Capture first: `grep … | tail` exits with *tail's* status, which
# is 0 even when grep matched nothing, so testing the pipeline
# directly would make the fallback dead code and print nothing at
# all for a log with no known signal — the exact case it exists for.
matched="$(grep -nE "$E2E_LOG_SIGNALS" "$log_file" | tail -n 60)"
if [ -n "$matched" ]; then
printf '%s\n' "$matched"
else
echo "(no signal lines matched — raw tail)"
tail -n 60 "$log_file" || true
fi
echo
done
echo "::endgroup::"
}
# A failing spec reports what it could not see; the renderer console
# reports why. `afterTest` already captures it — one `.console.log` per
# failing test — but only into the uploaded artifact, which a triager
# reading the job log never opens. That gap cost three days on #1925:
# `automation-trigger` said "the scheduled run's prompt never rendered"
# while the console said `Base branch "work" does not exist in this
# repository`, and only the second sentence names a fix.
#
# SEVERE only, and capped hard on both axes — this rides inside the
# digest, whose whole value is being small enough to survive the log
# tail. The artifact stays the complete record.
summarise_renderer_console() {
local artefacts='e2e-failure-artifacts' severe
[ -d "$artefacts" ] || return 0
severe="$(grep -h '"level":"SEVERE"' "$artefacts"/*.console.log 2>/dev/null | tail -n 15 || true)"
[ -n "$severe" ] || return 0
echo
echo "renderer console (SEVERE, last 15):"
# Long lines are usually a stack trace after the message; the first
# 300 characters carry the error itself — but only once the
# boilerplate in front of it is gone. Every line opens with a
# `file:///…/dist/renderer/app.js 292986:14 ` source location, and
# anything thrown through `ipcMain.handle` adds an `Error invoking
# remote method 'x:y': ` wrapper: together ~150 characters that say
# nothing and push the cause off the end. On run 33031502487 this
# printed `submodules unsupported (/opt/runner/_work` and stopped
# one character before the directory it had found — the single fact
# the whole digest existed to report.
printf '%s\n' "$severe" \
| sed -E "s#file://[^ ]+ [0-9]+:[0-9]+ ##; s#Error invoking remote method '[^']+': ##" \
| cut -c1-300
}
# The last thing the job prints, and deliberately not inside a group.
#
# Everything above competes for the job-log API's ~5000-line tail; this
# does not, because it is last and it is small. It re-states the two
# facts a triager always wants first — how many specs passed, and which
# ones failed with what — so they survive however large the dumps get.
summarise_e2e_attempt() {
local attempt="$1" run_log="$2"
[ -f "$run_log" ] || return 0
echo "── shard ${{ matrix.shard }} attempt ${attempt} digest ─────────────────"
grep -E '^Spec Files:' "$run_log" | tail -n 1 || true
# The reporter prints a `» <spec>` header for every file it runs, so
# keeping them all buried the one line that mattered under twenty
# that did not (run 31243627118 shard 2 printed all 21 paths for a
# single failing spec). And the error detail carries the same
# `[chrome …]` prefix as everything else, so anchoring on leading
# whitespace matched none of it — the digest named the spec but never
# said what failed.
#
# Select by worker instead: take the prefixes that reported a
# failure, then print those workers' lines whole. Header, verdict and
# error arrive together, whatever shape the error takes.
local failing
failing="$(grep -oE '^\[[^]]+\] [0-9]+ failing' "$run_log" | grep -oE '^\[[^]]+\]' | sort -u)"
if [ -n "$failing" ]; then
# Name every failing spec before printing any detail. The detail is
# capped, and a cap measured in lines drops whole workers once more
# than one spec fails: run 31266338302 shard 6 reported "3 failed"
# and named exactly one, because `tail -n 40` reached back only as
# far as the last worker's block. The names are what a reader needs
# first and they cost one line each, so they are never truncated.
#
# The raw wdio output lives in $run_log, which is not uploaded and
# not in the job log, so anything this function drops is gone.
echo "failing specs:"
printf '%s\n' "$failing" | grep -F -f - "$run_log" | grep -F '» ' | sort -u || true
echo
printf '%s\n' "$failing" | grep -F -f - "$run_log" | tail -n 120 || true
summarise_renderer_console
else
# No worker reported a failure, so the shard died some other way —
# a timeout, or a session that never came up. The raw tail is all
# there is, and printing nothing here is how a novel failure mode
# becomes invisible.
echo "(no per-spec failure reported — raw tail)"
tail -n 30 "$run_log" || true
fi
echo "──────────────────────────────────────────────────────────"
}
trap cleanup_e2e_processes EXIT
# Two attempts, not three: attempt 2 absorbs the flaky-session-startup
# case this retry exists for, while a deterministic failure (e.g. a
# selector that can never match) previously burned three full 480s
# attempts before reporting. Halves the cost of a genuine red shard.
for attempt in 1 2; do
cleanup_e2e_processes
run_log="e2e-failure-artifacts/attempt-${attempt}.log"
mkdir -p e2e-failure-artifacts
echo "::group::e2e shard ${{ matrix.shard }} attempt $attempt"
# Tee so the reporter output can be re-read for the digest below.
# PIPESTATUS, not $?, or we would read tee's status instead.
${TIMEOUT:+$TIMEOUT -k 15 480} npm run test:e2e:ci -- $SHARD_ARGS $SPEC_ARGS 2>&1 \
| tee "$run_log"
attempt_status=${PIPESTATUS[0]}
echo "::endgroup::"
if [ "$attempt_status" -eq 0 ]; then
exit 0
fi
# Order matters as much as volume: the job-log API returns the tail,
# so the biggest output goes first and the smallest, most useful
# output goes last. Bulk dumps, then diagnostics, then the digest.
dump_e2e_session_logs "$attempt" || true
capture_e2e_runner_diagnostics "$attempt" "$attempt_status" || true
summarise_e2e_attempt "$attempt" "$run_log" || true
echo "e2e shard ${{ matrix.shard }} attempt $attempt failed or timed out"
done
exit 1
# The specs above call saveScreenshot as they run, so this shard's slice has
# already re-rendered its reference PNGs into tests/e2e/screenshots/. Collect
# whatever changed (vs the committed shots) and hand it to the immutable
# candidate collector — no second e2e suite needed. PR-only; the sentinel keeps the artifact
# non-empty when a shard rendered no diffs.
- name: Collect changed reference screenshots
if: success() && github.event_name == 'pull_request'
run: |
mkdir -p changed/tests/e2e/screenshots
touch "changed/.shard-${{ matrix.shard }}"
# Untracked PNGs are first-time reference shots from a new spec and
# must be included alongside modifications.
mapfile -t files < <(
{
git diff --name-only -- tests/e2e/screenshots/
git ls-files --others --exclude-standard -- tests/e2e/screenshots/
} | sort -u
)
for f in "${files[@]}"; do
[ -n "$f" ] || continue
cp "$f" "changed/tests/e2e/screenshots/"
done
echo "Collected ${#files[@]} changed screenshot(s) for shard ${{ matrix.shard }}"
- uses: actions/upload-artifact@v7
if: success() && github.event_name == 'pull_request'
with:
name: screenshots-shard-${{ matrix.shard }}
path: changed/
include-hidden-files: true
if-no-files-found: error
# Only consumed by screenshot-artifacts within this same run; it never
# needs to outlive the run. Uploaded per shard on every PR, so the
# default 90-day retention made this the largest draw on the org's
# Actions storage quota. Expire fast, keeping a small debug window.
retention-days: 3
# On failure, surface the screenshot + page source captured in afterTest,
# plus the per-attempt cgroup/process snapshot above. The latter is still
# available when ChromeDriver is too dead to capture browser artifacts.
# Skipped only when a hard runner OOM kills the whole job before this step.
- uses: actions/upload-artifact@v7
if: failure()
continue-on-error: true
with:
name: e2e-failure-artifacts-shard-${{ matrix.shard }}
path: e2e-failure-artifacts/
if-no-files-found: ignore
retention-days: 3
# Preserve reference screenshots as immutable run evidence. Each e2e shard
# uploads only PNGs that differ from the checked-out baseline; this job merges
# those slices into one downloadable candidate set. It deliberately has no
# write permission and never checks out or pushes the PR branch. The trusted
# publish-screenshot-candidates workflow consumes the finished artifact, opens
# a bot-owned child PR into the source branch, and links it from the parent for
# explicit image-diff review. Forks retain the artifact-only/manual path.
screenshot-artifacts:
needs: [build, e2e]
if: >-
always() && needs.build.result == 'success' && needs.e2e.result == 'success' &&
github.event_name == 'pull_request'
runs-on: ${{ vars.SELF_HOSTED_CHECKS || 'ubuntu-latest' }}
# Raised from 10 to cover the checkout + dependency install that the noise
# filter below needs. Both are cache-warm on the fleet; the budget is for a
# cold runner, not the steady state.
timeout-minutes: 20
permissions:
contents: read
steps:
# Checkout FIRST: `actions/checkout` cleans the workspace, so downloading
# the shard artifacts before it would wipe them. No `ref:` — the default
# merge commit is what the e2e shards rendered against, so the filter's
# `git diff` reproduces exactly the comparison they made. `fetch-depth: 0`
# for the filter's flap lookback and its `origin/main` scope base.
- uses: actions/checkout@v7.0.1
with:
fetch-depth: 0
# Mirrors precheck: never leave even the read-only GITHUB_TOKEN in a
# fork checkout, but keep it for same-repo runs, which is the only
# shape `./.github/actions/setup` is proven against in this pipeline.
persist-credentials: ${{ github.event.pull_request.head.repo.full_name == github.repository }}
- uses: ./.github/actions/setup
- name: Download rendered screenshot candidates
uses: actions/download-artifact@v8
with:
pattern: screenshots-*
merge-multiple: true
path: candidates
# Reference PNGs are pixel-rendered on the runner, so a re-render of an
# UNCHANGED screen still differs by a handful of anti-aliased pixels. Left
# unfiltered every PR ships a review PR of near-wholesale re-renders —
# measured at 280 shots on #2134, a PR that changed no source file at all,
# of which `filter-screenshots` classified 0 as real. Reviewers cannot find
# signal in that, so they rubber-stamp it.
#
# `scripts/filter-screenshots.mts` (#609) already encodes the keep /
# ignore / flap decision and was written for exactly this, but it was only
# ever documented as a manual local step. Run it here, in the job that
# merges the shard slices, so the candidate artifact carries only real
# changes and the downstream review PR is worth opening.
- name: Drop render noise from the candidate set
env:
# The filter's documented escape hatch: the label means "refresh
# everything", including sub-threshold micro-diffs.
UPDATE_SCREENSHOTS_LABEL: >-
${{ contains(github.event.pull_request.labels.*.name, 'update-screenshots') }}
run: |
set -euo pipefail
shots='candidates/tests/e2e/screenshots'
if ! compgen -G "$shots/*.png" > /dev/null; then
echo 'No candidate PNGs to filter.'
exit 0
fi
# Lay the re-renders over the checked-out baseline. A plain copy, not
# `git checkout` — the latter writes the index too, and the filter
# reads the UNSTAGED working-tree diff (`git diff --name-only`), so a
# staged candidate is invisible to it and would be silently kept.
cp "$shots"/*.png tests/e2e/screenshots/
# Restores every sub-threshold shot to its committed bytes in place.
node scripts/filter-screenshots.mts
# Whatever the filter restored no longer differs from the baseline, so
# drop it from the artifact. Iterating the artifact (not the worktree)
# keeps the per-shard `.shard-N` provenance markers untouched.
for png in "$shots"/*.png; do
rel="tests/e2e/screenshots/$(basename "$png")"
if git diff --quiet -- "$rel" 2>/dev/null &&
! git ls-files --others --exclude-standard --error-unmatch "$rel" > /dev/null 2>&1
then
rm -f "$png"
fi
done
- name: Summarize candidate screenshots
id: candidates
run: |
mapfile -t files < <(
find candidates/tests/e2e/screenshots -type f -name '*.png' 2>/dev/null | sort
)
echo "count=${#files[@]}" >> "$GITHUB_OUTPUT"
{
echo "## E2E screenshot candidates"
echo
if [ ${#files[@]} -eq 0 ]; then
echo "All rendered screenshots match the committed references."
else
echo "${#files[@]} changed reference screenshot(s) are attached as an immutable artifact."
echo
echo "A trusted follow-up workflow will put these PNGs on a bot-owned branch,"
echo "open a child PR into the source branch, and link it from the parent for review."
echo "The artifact remains the immutable record and the manual fallback for forks."
echo
printf -- '- `%s`\n' "${files[@]#candidates/}"
fi
} >> "$GITHUB_STEP_SUMMARY"
- name: Upload immutable screenshot candidates
if: steps.candidates.outputs.count != '0'
uses: actions/upload-artifact@v7
with:
name: reference-screenshot-candidates-${{ github.run_id }}
# Preserve the repository-relative tree so extracting over a checkout
# places the PNGs exactly where the reviewer expects.
path: candidates/
if-no-files-found: error
retention-days: 14
# Single aggregate gate. Require ONLY this check in branch protection: it
# `needs` every other job (and e2e's result aggregates all 8 shards), so it
# turns the whole pipeline into one green/red status. New jobs or e2e shard
# counts can change without touching the required-checks list — just add the
# job to `needs` here. `if: !cancelled()` so this runs even when an upstream
# job fails/is skipped; we then assert none failed or were cancelled. Fork PRs
# have their own explicit safe-tier result below: they may pass only after
# `precheck` actually ran on a GitHub-hosted runner, never merely because all
# code-executing jobs were skipped.
#
# Concurrency supersession: top-level `cancel-in-progress` cancels in-flight
# jobs when a newer push/sync arrives on the same ref. Those cancelled jobs
# must not paint the superseded SHA's `CI Passed` red — tip CI is the gate
# that matters. Two different mechanisms cover that, and which one applies
# depends on whether the RUN was cancelled or only some of its JOBS were:
#
# - Whole run cancelled (concurrency supersession, manual cancel): this job
# is SKIPPED by the `if:` below and never reports. The superseding run's
# own `CI Passed` is the one that counts.
# - Individual jobs cancelled while the run itself lives on (skip-mode's
# check/build/e2e — issue #609 — matrix cancels, a runner losing
# communication): this job still runs, and the `ANY_CANCELLED` branch in
# the step below decides. It detects supersession by (1) comparing this
# run's head SHA to the live tip, and (2) looking for a newer CI run on
# the same head SHA (e.g. a delayed nightly schedule sharing
# `ci-refs/heads/main` with a tip push). A job-level cancel still at tip
# with no newer run fails. Sibling-matrix cancels after a real shard
# failure fail because `ANY_FAILURE` is true first.
#
# KNOWN CONSEQUENCE of the first bullet: a skipped required check satisfies
# branch protection, so a manually cancelled run now reads as satisfied rather
# than red. That is a deliberate trade for unwedging the concurrency group
# (see the `if:` below); the case it costs is a human cancelling a run they
# do not intend to re-run, which the next push re-gates anyway.
ci-passed:
# A push to trunk (`main`) intentionally skips the expensive tier. Give that
# run a distinct check context so its green result on the same head SHA
# cannot satisfy `release`'s required `CI Passed` check before promotion e2e
# has run.
#
# The literal name stays `Develop CI Passed` even though trunk is no longer
# called `develop`: it is a check-context string that branch protection may
# match on by name, so renaming it is a separate change that has to land
# together with the protection rule.
#
# Fork PRs get their own context for the same reason, and it closes the last
# of #800's acceptance criteria: "branch protection distinguishes 'tested and
# passed' from 'not executed'". A fork run is genuinely green — `precheck`
# executes the fork's typecheck and lint on a hosted runner, and gitleaks.yml
# scans it — but `check`, `build` and `e2e` never dispatch, so unit tests, the
# bundle and the whole e2e tier are unrun. Publishing that under the same
# `CI Passed` string as a fully-tested same-repo run made the two
# indistinguishable to branch protection and to anyone reading the checks
# list. Now a fork PR simply has no `CI Passed` check, so a rule requiring one
# holds the merge until a maintainer takes the change through a same-repo
# branch — the same mechanism that stops a trunk push satisfying `release`.
#
# Additive on purpose: same-repo PRs still publish `CI Passed` byte-for-byte,
# so no existing protection rule or consumer changes meaning. (`release-mac.yml`
# matches this name for a release SHA, which is never a fork PR.)
name: >-
${{ (github.event_name == 'pull_request' &&
github.event.pull_request.head.repo.full_name != github.repository) && 'Fork CI Passed' ||
github.event_name == 'push' && github.ref == 'refs/heads/main' && 'Develop CI Passed' ||
'CI Passed' }}
# NOT `always()`. That is true even when the RUN is cancelled, so GitHub
# still creates and queues this job on a run that is being torn down — and
# `runs-on` below prefers the self-hosted fleet, where it can then wait for
# a runner that is never coming. A queued job keeps its run out of a
# terminal state, the run keeps holding the `ci-<pr>` concurrency group, and
# the run that superseded it therefore never starts: it sits `pending` with
# zero jobs created. Observed on #1669, where promoting a draft cancelled
# the in-flight run and CI was wedged for 114 minutes — until a fleet runner
# freed up and the orphan finally ran, correctly reporting "superseded" in
# about five seconds. That is the point: the job was never broken, it just
# should not have been queued for a runner at all on a run nobody was
# waiting for. Meanwhile a normal cancel is accepted (202) but does not
# dequeue it, and `force-cancel` is the only manual way out.
#
# `!cancelled()` keeps everything `always()` was buying — this still runs
# when an upstream job FAILS or is SKIPPED, which the default `success()`
# would not — and differs only on a cancelled run, where being skipped is
# what lets the run finish and release the group. GitHub documents this
# exact substitution as the fix for `always()` hangs.
if: ${{ !cancelled() }}
needs: [precheck, check, bench, build, e2e, screenshot-artifacts]
# Tiny aggregate gate. Hosted by default; SELF_HOSTED_CHECKS=copse-checks
# opts it onto the self-hosted check fleet. Fails closed either way: if the fleet
# is saturated this stays pending (never green), so it can't wave a run
# through without its required status check.
#
# "Stays pending" is the right answer for a LIVE run and the wrong one for a
# cancelled run, where it wedges the concurrency group — which is why the
# `if:` above must never go back to `always()` while this prefers the fleet.
# Pinning this one job to `ubuntu-latest` is the other way to buy that (it
# would start in seconds and could never queue behind the fleet), at roughly
# one billed hosted minute per run; deliberately not taken.
#
# Fork PRs go to GitHub-hosted regardless: fork runs must stay off the
# fleet entirely, and the fleet's runner group should not serve public-repo
# jobs anyway — routing there would queue this gate forever.
runs-on: ${{ (github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name != github.repository) && 'ubuntu-latest' || vars.SELF_HOSTED_CHECKS || 'ubuntu-latest' }}
timeout-minutes: 15
permissions:
contents: read
pull-requests: read
actions: read
steps:
- name: Check upstream job results
env:
GH_TOKEN: ${{ github.token }}
REPOSITORY: ${{ github.repository }}
EVENT_NAME: ${{ github.event_name }}
REF_NAME: ${{ github.ref_name }}
RUN_HEAD_SHA: ${{ github.event.pull_request.head.sha || github.sha }}
PR_NUMBER: ${{ github.event.pull_request.number || '' }}
CURRENT_RUN_ID: ${{ github.run_id }}
# Self-hosted check-fleet images ship curl+jq but not the gh CLI
# (ci-runners/Dockerfile). #1017 moved this gate back onto that fleet,
# so tip/supersession lookups must use the REST API — `gh api` fails
# with `gh: command not found` and the gate fail-closes on every
# concurrency cancel (false red on main).
GITHUB_API_URL: ${{ github.api_url }}
run: |
# Thin REST helper — curl+jq are on both hosted and self-hosted images.
api_get() {
curl -fsS \
-H "Authorization: Bearer ${GH_TOKEN}" \
-H "Accept: application/vnd.github+json" \
-H "X-GitHub-Api-Version: 2022-11-28" \
"${GITHUB_API_URL}/$1"
}
FORK_PR=${{ github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name != github.repository }}
if $FORK_PR; then
if [ "${{ needs.precheck.result }}" != "success" ]; then
echo "Fork PR safe check tier was not executed successfully."
echo '${{ toJSON(needs) }}'
exit 1
fi
echo "Fork PR safe check tier passed (typecheck, lint, format, API protocol)."
echo "Unit tests, build and e2e were intentionally not dispatched: fork code"
echo "must not run on this repository's runners. This run therefore reports as"
echo "'Fork CI Passed', not 'CI Passed' — it is not evidence the change is tested."
exit 0
fi
MODE="${{ needs.precheck.outputs.mode }}"
ANY_FAILURE=${{ contains(needs.*.result, 'failure') }}
ANY_CANCELLED=${{ contains(needs.*.result, 'cancelled') }}
CORE_OK=${{ needs.precheck.result == 'success' }}
# Skip-mode runs (human-reviewed screenshot-only commits) intentionally
# do not execute check/build/e2e;
# those jobs surface as 'cancelled', which must not fail the gate —
# that painted every bot push red and blocked auto-merge (issue
# #609). Genuine failures still fail even in skip mode.
if [ "$MODE" = "skip" ]; then
if $ANY_FAILURE || ! $CORE_OK; then
echo "Skip-mode run has a genuine failure:"
echo '${{ toJSON(needs) }}'
exit 1
fi
echo "All CI jobs passed (heavy jobs intentionally skipped: mode=skip)."
exit 0
fi
# Demand e2e whenever the job's own dispatch contract says a
# merge-eligible same-repository PR must run it. Mirroring the draft
# and zero-shard guards avoids demanding a deliberately skipped job.
E2E_REQUIRED=${{ github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name == github.repository && (github.event.pull_request.draft == false || contains(github.event.pull_request.labels.*.name, 'ci-full')) }}
if $E2E_REQUIRED && [ "${{ needs.precheck.outputs.e2e_shard_total }}" != "0" ] && [ "${{ needs.e2e.result }}" != "success" ]; then
echo "Required PR e2e did not complete successfully (result=${{ needs.e2e.result }})."
echo '${{ toJSON(needs) }}'
exit 1
fi
if $ANY_FAILURE; then
echo "One or more CI jobs failed:"
echo '${{ toJSON(needs) }}'
exit 1
fi
if $ANY_CANCELLED; then
# Fail closed if we cannot resolve tip — a tip cancel must stay red.
if [ -n "$PR_NUMBER" ]; then
TIP_SHA=$(api_get "repos/${REPOSITORY}/pulls/${PR_NUMBER}" | jq -er .head.sha) || {
echo "Could not resolve PR tip SHA; treating cancel as failure."
echo '${{ toJSON(needs) }}'
exit 1
}
else
ENC_REF=$(printf '%s' "$REF_NAME" | jq -sRr @uri)
TIP_SHA=$(api_get "repos/${REPOSITORY}/commits/${ENC_REF}" | jq -er .sha) || {
echo "Could not resolve branch tip SHA; treating cancel as failure."
echo '${{ toJSON(needs) }}'
exit 1
}
fi
if [ -n "$TIP_SHA" ] && [ "$TIP_SHA" != "$RUN_HEAD_SHA" ]; then
echo "Run superseded by newer tip ${TIP_SHA} (this SHA=${RUN_HEAD_SHA}); cancelled jobs are concurrency noise."
echo '${{ toJSON(needs) }}'
exit 0
fi
# Same tip, but a newer CI run on this SHA may have taken the
# concurrency slot (delayed schedule, re-run, workflow_dispatch).
NEWER_RUN_ID=$(api_get \
"repos/${REPOSITORY}/actions/workflows/ci.yml/runs?head_sha=${RUN_HEAD_SHA}&per_page=20" \
| jq -r "[.workflow_runs[] | select(.id > ${CURRENT_RUN_ID}) | .id][0] // empty") || {
echo "Could not resolve newer workflow runs; treating cancel as failure."
echo '${{ toJSON(needs) }}'
exit 1
}
if [ -n "$NEWER_RUN_ID" ]; then
echo "Run superseded by newer CI run ${NEWER_RUN_ID} on tip ${RUN_HEAD_SHA}; cancelled jobs are concurrency noise."
echo '${{ toJSON(needs) }}'
exit 0
fi
echo "One or more CI jobs were cancelled while still at tip (${RUN_HEAD_SHA}):"
echo '${{ toJSON(needs) }}'
exit 1
fi
echo "All CI jobs passed (or were intentionally skipped)."