Skip to content

fix(ci): serialize GLM-backed E2E harnesses to reduce backend contention#61

Closed
nhuelstng wants to merge 1 commit into
mainfrom
fix/e2e-concurrent-backend-load
Closed

fix(ci): serialize GLM-backed E2E harnesses to reduce backend contention#61
nhuelstng wants to merge 1 commit into
mainfrom
fix/e2e-concurrent-backend-load

Conversation

@nhuelstng

Copy link
Copy Markdown
Contributor

What

Serializes opencode/codex/copilot (one at a time, per OS) in the E2E workflow's concurrency group, while leaving claude-code free to run in parallel.

Why

Triggering the E2E workflow on #60 (run 29240222298) failed 6 of 7 matrix jobs with agent did not exit within 5m0s. Investigation (full notes below) shows:

  • omac's own instrumented paths (supervisor, sandbox launcher, netproxy filter — including the new audit-trail Emit() calls added in 37 add security audit trail #38) return promptly in every local repro I tried, including one that drives real proxied network traffic (allow + deny) through the sandbox. No hang reproduces there.
  • In the failing CI runs, omac's own log shows the CLI's single ALLOW external.model.tngtech.com:443 decision at session start and then nothing else until the 5-minute deadline fires — the harness process (copilot/opencode/codex) had already printed its final answer/summary and simply never returned control.
  • The one job that passed cleanly every time (claude-code) is also the only harness that does not share the SKAINET_TOKEN-backed GLM-5.2 endpoint (external.model.tngtech.com) — opencode, codex, and copilot all hit that same backend, and the workflow runs them fully concurrently.
  • This exact failure signature (agent did not exit within 5m0s, across the whole matrix) already occurred in a scheduled run on main on 2026-07-10 — three commits before the audit-trail commit (37 add security audit trail #38) ever merged. That rules out 37 add security audit trail #38, and rules out anything in fix(e2e): read audit-output.txt in TestE2EProvenance instead of chat text #60, as the cause.

Net: this looks like contention/stalling on the shared GLM-5.2 backend under concurrent load (Node-based CLIs sitting on an open connection rather than exiting), not a code defect in this repo.

How

Change the job's concurrency.group so the three GLM-backed harnesses share a single group per OS (serialized, one at a time), while claude-code keeps its own independent group and still runs in parallel.

Verification

  • python3 -c "import yaml; yaml.safe_load(open('.github/workflows/e2e.yml'))" — valid YAML
  • Manually checked the contains(fromJSON(...), matrix.harness) expression against GitHub Actions' documented expression functions (both are valid in any expression context, including concurrency.group)
  • Not proven: this is a mitigation for a suspected environmental/backend cause, not a confirmed fix — I could not reproduce the hang locally to verify the fix closes it, since it depends on live shared-backend load. Recommend watching the next few scheduled/dispatched E2E runs for whether the hang recurs.

Related: #60 (separate fix for the TestE2EProvenance flake surfaced by the same investigation).

The E2E run triggered on PR #60 (run 29240222298) hung on 6 of 7 matrix
jobs with "agent did not exit within 5m0s" — the CLI process never
returned after finishing its response, and omac's own logs show zero
further activity, so the process is stuck outside omac's code. The one
job that passed (claude-code) is also the only harness that doesn't
share the SKAINET_TOKEN-backed GLM-5.2 endpoint with opencode/codex/
copilot; those three ran fully concurrently against the same backend.
This exact failure signature predates the audit-trail commit (seen on
main on 2026-07-10), ruling out a regression in this repo's code.

Serialize the three GLM-backed harnesses per OS via a shared
concurrency group, leaving claude-code free to run in parallel since it
talks to a separate backend. This is a mitigation for suspected shared-
backend contention, not a proven fix — it can't be conclusively
verified without more E2E runs.

Signed-off-by: Niclas Hülsmann <niclas.huelsmann@tngtech.com>
@nhuelstng nhuelstng closed this Jul 13, 2026
@nhuelstng nhuelstng deleted the fix/e2e-concurrent-backend-load branch July 13, 2026 11:19
@nhuelstng

Copy link
Copy Markdown
Contributor Author

this was fixed by #60 and #62

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant