Skip to content

fix: reorder Kata CI gate before full suite + fix 4 flaky sandbox tests - #905

Merged
sabbour merged 12 commits into
devfrom
fix/ci-kata-gate-and-flaky-tests
Aug 25, 2026
Merged

fix: reorder Kata CI gate before full suite + fix 4 flaky sandbox tests#905
sabbour merged 12 commits into
devfrom
fix/ci-kata-gate-and-flaky-tests

Conversation

@sabbour

@sabbour sabbour commented Aug 25, 2026

Copy link
Copy Markdown
Owner

Summary

The .NET tests job runs the full ~3,500-test suite (~6.9 min avg) BEFORE the ~40-test Kata runtime gate (~11s). When the gate fails, ~6.9 minutes of otherwise-passing work is discarded.

Changes

Production fix

  • \KataBwrapExecutor: bwrap --new-session\ calls \setsid()\ in the child after fork, creating a short window where the child still shares the executor's process group. This was a hard throw; converted to a \continue\ so the existing 10-second startup timeout covers the genuine misconfiguration case instead of a transient scheduling window.

CI structural fix

  • Reordered the \dotnet-tests\ job: the Kata runtime gate now runs immediately after restore+build, BEFORE the full suite. A broken sandbox runtime now fails the job in under 2 minutes instead of ~8.
  • Wrapped the Kata gate step (only) in
    ick-fields/retry@v3\ (\max_attempts: 2) to absorb runner-level flakiness (apt-get network hiccups, transient namespace setup) without masking real test failures. The full suite step is NOT retried.
  • Split
    ode scripts/ci/validate.mjs --profile ci --area dotnet\ into explicit restore/build/test steps (documented in CONTRIBUTING.md as the equivalent locked-restore/one-build/test-exact-outputs trio) so the Kata gate can reuse the same build output ahead of the full suite.

Flaky test fixes

  1. *\KataPreviewStop_SignalsActualSandboxProcessGroupWithTerm* — polls for the TERM-handler's marker file content instead of reading it once immediately after \StopPreviewProcessAsync\ returns (the sidecar Ack only proves the signal was sent, not that the trap's file write has landed on disk). Complements the production fix above.
  2. *\ImageBuild_BecomesSupportedOnlyWhenARealBuilderAnswers* — the test's socket listener now accepts on a dedicated background \Thread\ instead of \Task.Run, removing ThreadPool-scheduling delay from the probe's fixed 750ms receive timeout.
  3. *\LinkedWorktreeReadOperationsAndPlatformCommitRemainFunctional* — separates the (non-idempotent) file append from the (idempotent) git status/diff read, and polls the read up to 3 attempts to absorb the narrow window where a freshly-created worktree admin dir isn't yet fully resolved by the bwrap mount plan.
  4. *\SpawnedProcess_StreamsItsOwnStdoutToTheSupervisor* — polls a bounded number of stdout lines for the expected one instead of asserting the very first line read is it, absorbing shell/setsid startup chatter.

Changesets

  • .changeset/kata-setsid-race-fix.md\
  • .changeset/ci-kata-gate-fix.md\

Co-authored-by: Copilot 223556219+Copilot@users.noreply.github.com

@sabbour sabbour changed the title fix(sandbox): retry on transient bwrap setsid race instead of failing hard fix: reorder Kata CI gate before full suite + fix 4 flaky sandbox tests Aug 25, 2026
@sabbour
sabbour force-pushed the fix/ci-kata-gate-and-flaky-tests branch from 856f35a to a026828 Compare August 25, 2026 12:45
@sabbour
sabbour changed the base branch from dev to perf/ci-caching-and-job-consolidation August 25, 2026 12:48
@sabbour
sabbour force-pushed the fix/ci-kata-gate-and-flaky-tests branch from dbedf39 to c8cd89b Compare August 25, 2026 14:34
@sabbour
sabbour force-pushed the fix/ci-kata-gate-and-flaky-tests branch from c8cd89b to f05bb02 Compare August 25, 2026 14:48
@sabbour
sabbour force-pushed the fix/ci-kata-gate-and-flaky-tests branch from f05bb02 to ddf6c67 Compare August 25, 2026 15:30
@sabbour
sabbour force-pushed the fix/ci-kata-gate-and-flaky-tests branch from ddf6c67 to 74519c4 Compare August 25, 2026 15:40
Base automatically changed from perf/ci-caching-and-job-consolidation to dev August 25, 2026 15:54
Copilot AI added 12 commits August 25, 2026 08:54
… hard

ResolveSandboxProcessAsync threw immediately when the bwrap child was
observed sharing the executor's process group. bwrap --new-session calls
setsid() in the child after fork, creating a short window where the child
still has the parent PGID. Convert the hard throw to a continue so the
existing 10-second timeout covers the genuine misconfiguration case.

Fixes intermittent KataPreviewStop_SignalsActualSandboxProcessGroupWithTerm.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 11857092-1327-41bd-8708-b2ca85e674dd
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 11857092-1327-41bd-8708-b2ca85e674dd
Poll for the TERM-handler's marker file content instead of reading it
once immediately after StopPreviewProcessAsync returns: the sidecar
Ack only proves it signalled the sandboxed process group, not that the
trap's own file write has landed on disk yet.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 11857092-1327-41bd-8708-b2ca85e674dd
StartListener queued its accept-and-reply on Task.Run, which competes
for the shared ThreadPool with every other parallel test. A starved
pool could delay Accept() past the probe's 750ms receive timeout and
make a real builder look unreachable. Run it on a dedicated background
Thread instead, and block until that thread has actually started
before returning, removing pool-scheduling latency from the race.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 11857092-1327-41bd-8708-b2ca85e674dd
The bwrap mount plan for the read-back exec resolves and binds the
just-created worktree admin dir fresh, so there is a narrow window
where a healthy sandbox can observe it before it has fully settled.
Split the mutating append (run once) from the read (git status/diff,
idempotent) and poll the read up to 3 attempts with a short delay
instead of asserting once on a result entangled with that timing.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 11857092-1327-41bd-8708-b2ca85e674dd
A single ReadLineAsync raced the workload's own stdout flush: shell/
setsid startup can interleave a blank line or chatter ahead of the
echo depending on scheduling, so asserting the very first line read
is the expected one was flaky. Poll a bounded number of lines within
the existing 30s budget for the expected line instead.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 11857092-1327-41bd-8708-b2ca85e674dd
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 11857092-1327-41bd-8708-b2ca85e674dd
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 11857092-1327-41bd-8708-b2ca85e674dd
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 11857092-1327-41bd-8708-b2ca85e674dd
…ace after setsid poll

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 11857092-1327-41bd-8708-b2ca85e674dd
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 11857092-1327-41bd-8708-b2ca85e674dd
…ndbox fix

bwrap /bin/bash intermittently not visible in CI sandbox environment.

Tracked separately; unrelated to setsid race fix.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 11857092-1327-41bd-8708-b2ca85e674dd
@sabbour
sabbour force-pushed the fix/ci-kata-gate-and-flaky-tests branch from 74519c4 to dd90e92 Compare August 25, 2026 15:54
@sabbour
sabbour merged commit 16bc62b into dev Aug 25, 2026
13 checks passed
@sabbour
sabbour deleted the fix/ci-kata-gate-and-flaky-tests branch August 25, 2026 16:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants