fix(ci): serialize GLM-backed E2E harnesses to reduce backend contention#61
Closed
nhuelstng wants to merge 1 commit into
Closed
fix(ci): serialize GLM-backed E2E harnesses to reduce backend contention#61nhuelstng wants to merge 1 commit into
nhuelstng wants to merge 1 commit into
Conversation
The E2E run triggered on PR #60 (run 29240222298) hung on 6 of 7 matrix jobs with "agent did not exit within 5m0s" — the CLI process never returned after finishing its response, and omac's own logs show zero further activity, so the process is stuck outside omac's code. The one job that passed (claude-code) is also the only harness that doesn't share the SKAINET_TOKEN-backed GLM-5.2 endpoint with opencode/codex/ copilot; those three ran fully concurrently against the same backend. This exact failure signature predates the audit-trail commit (seen on main on 2026-07-10), ruling out a regression in this repo's code. Serialize the three GLM-backed harnesses per OS via a shared concurrency group, leaving claude-code free to run in parallel since it talks to a separate backend. This is a mitigation for suspected shared- backend contention, not a proven fix — it can't be conclusively verified without more E2E runs. Signed-off-by: Niclas Hülsmann <niclas.huelsmann@tngtech.com>
Contributor
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Serializes
opencode/codex/copilot(one at a time, per OS) in the E2E workflow's concurrency group, while leavingclaude-codefree to run in parallel.Why
Triggering the E2E workflow on #60 (run 29240222298) failed 6 of 7 matrix jobs with
agent did not exit within 5m0s. Investigation (full notes below) shows:omac's own instrumented paths (supervisor, sandbox launcher, netproxy filter — including the new audit-trailEmit()calls added in 37 add security audit trail #38) return promptly in every local repro I tried, including one that drives real proxied network traffic (allow + deny) through the sandbox. No hang reproduces there.omac's own log shows the CLI's singleALLOW external.model.tngtech.com:443decision at session start and then nothing else until the 5-minute deadline fires — the harness process (copilot/opencode/codex) had already printed its final answer/summary and simply never returned control.claude-code) is also the only harness that does not share theSKAINET_TOKEN-backed GLM-5.2 endpoint (external.model.tngtech.com) —opencode,codex, andcopilotall hit that same backend, and the workflow runs them fully concurrently.agent did not exit within 5m0s, across the whole matrix) already occurred in a scheduled run onmainon 2026-07-10 — three commits before the audit-trail commit (37 add security audit trail #38) ever merged. That rules out 37 add security audit trail #38, and rules out anything in fix(e2e): read audit-output.txt in TestE2EProvenance instead of chat text #60, as the cause.Net: this looks like contention/stalling on the shared GLM-5.2 backend under concurrent load (Node-based CLIs sitting on an open connection rather than exiting), not a code defect in this repo.
How
Change the job's
concurrency.groupso the three GLM-backed harnesses share a single group per OS (serialized, one at a time), whileclaude-codekeeps its own independent group and still runs in parallel.Verification
python3 -c "import yaml; yaml.safe_load(open('.github/workflows/e2e.yml'))"— valid YAMLcontains(fromJSON(...), matrix.harness)expression against GitHub Actions' documented expression functions (both are valid in any expression context, includingconcurrency.group)Related: #60 (separate fix for the
TestE2EProvenanceflake surfaced by the same investigation).