Skip to content

fix(e2e): unblock onboarding agent's permissions, fix macOS timeout#97

Merged
nhuelstng merged 2 commits into
mainfrom
fix/e2e-readme-onboarding-auto-timeout
Jul 15, 2026
Merged

fix(e2e): unblock onboarding agent's permissions, fix macOS timeout#97
nhuelstng merged 2 commits into
mainfrom
fix/e2e-readme-onboarding-auto-timeout

Conversation

@nhuelstng

Copy link
Copy Markdown
Contributor

Summary

Second round of fixes after re-running e2e-readme-onboarding.yml on main post-#96:

  • ubuntu-latest: the fix(e2e): fix README-onboarding script for macOS bash 3.2 and runner XDG dirs #96 fix worked (opencode read the isolated config, resolved GLM-5.2 correctly), and the agent actually started — read README.md, made a plan, began investigating the environment. It then hit a wall: opencode's non-interactive run mode auto-rejects any permission request it can't ask about interactively, including reading paths outside the workdir (/etc/os-release) or invoking package managers — i.e. everything this test needs the agent to do.
  • macos-latest: failed immediately with timeout: command not found — macOS ships no GNU coreutils timeout, and its BSD sleep doesn't accept duration suffixes like 20m either.

Why

  • --auto is opencode's documented equivalent of claude-code's --dangerously-skip-permissions (opencode run --help: "auto-approve permissions that are not explicitly denied (dangerous!)") — same rationale as that flag in claudeCodeConfig() (internal/e2e/harnesses.go), and inherent to this test's premise of giving the agent a genuinely normal, unrestricted shell.
  • timeout needed a portable replacement since only Linux runners have it.

How

  • Added --auto to the opencode run invocation.
  • Replaced timeout --signal=TERM with a small run_with_timeout() bash function (background the command, race a sleep+kill watchdog, wait), and switched the timeout value from a duration string (20m) to plain seconds so sleep works identically under GNU and BSD.
  • Caught a bug in my own first draft of run_with_timeout before it shipped: it toggled set -e/set +e inside the function, but those are global, not function-scoped — it silently re-armed errexit right before returning, which would abort the whole script on a non-zero agent exit instead of letting the caller read agent_status. Fixed by capturing the status via wait "$pid" || status=$? instead of toggling errexit at all.

Verification

  • shellcheck clean; syntax-checked on both a modern bash and the real bash:3.2 image.
  • Verified run_with_timeout directly against three cases (quick success, quick failure, real timeout-kill) on both bash versions, confirming correct exit-status propagation in each — including reproducing and then fixing the set -e leak bug above before it reached CI.
  • Not able to fully verify --auto actually lets the agent complete a real install locally (that would mean running an unrestricted, --auto-approved agent session against this dev machine, which is out of scope — the whole point of --auto is to remove guardrails, and that should only happen on the disposable CI runner this test targets). Confirming this requires another real workflow run.

Second manual run (after the bash-3.2/XDG fix) got further - opencode
resolved config/model correctly this time and started reading the
README - but hit two more real bugs:

- opencode's non-interactive `run` mode auto-REJECTS any permission
  request it can't ask about interactively, including reading paths
  outside the workdir (e.g. /etc/os-release) or running package
  managers. That's everything this test needs the agent to do. Add
  --auto (opencode's equivalent of claude-code's
  --dangerously-skip-permissions) so it can actually act.

- macOS has no `timeout` binary (GNU coreutils only) and its BSD sleep
  rejects duration suffixes like "20m" - the macOS leg failed
  immediately with "timeout: command not found". Replace it with a
  small portable run_with_timeout() (background + sleep-then-kill
  watchdog + wait), and switch the timeout value to plain seconds so
  `sleep` works on both GNU and BSD. Verified against both a modern
  bash and the real bash:3.2 image, including the case where the
  wrapped command exits well before the timeout - an earlier draft of
  this wrapper toggled `set -e`/`set +e` globally inside the function,
  which leaked out and re-armed errexit before the caller could read
  $?, aborting the script early; fixed by capturing the status via
  `wait "$pid" || status=$?` instead.

Signed-off-by: Niclas Hülsmann <niclas.huelsmann@tngtech.com>
…t flip the exit code

Both OS legs of the third manual run actually succeeded end-to-end this
time (agent installed omac, got doctor healthy, launched omac start,
and wrote a real report) - but ubuntu-latest still reported red.

Root cause: the agent's own (documented, README-driven) `go install`
attempt populates the Go module cache under $DRIVER_HOME/go/pkg/mod,
which Go deliberately makes read-only. The script's `trap 'rm -rf
"$WORK"' EXIT` then fails with a wall of "Permission denied", and
since that's the EXIT trap's own exit status, it silently overrides
whatever exit code the script had already decided on - turning a real
pass into a false failure. Reproduced and fixed locally: chmod
everything writable before rm -rf.

Signed-off-by: Niclas Hülsmann <niclas.huelsmann@tngtech.com>
@nhuelstng

Copy link
Copy Markdown
Contributor Author

Pushed one more fix and re-ran directly on this branch until both legs went green with real, substantive sessions:

  • Run 1 (29409220161): macOS passed genuinely (5m22s real session — installed via Homebrew, found the brew trust tap-authorization gap, launched omac start). Ubuntu's agent also fully succeeded (installed via .deb, fixed a real AppArmor/bwrap gap using doctor's own guidance, launched omac start) — but the job still went red.
  • Root cause: the agent's go install attempt (part of its real, README-driven exploration) populates the Go module cache read-only by design, and the script's trap 'rm -rf "$WORK"' EXIT cleanup then fails with "Permission denied" — which, being the EXIT trap's own status, silently overrode the script's already-correct success exit code. Fixed by chmod -R u+w before the rm -rf, verified locally that this restores the intended exit code without leaking the trap's own failure.
  • Run 2 (29409691842): both legs green, ~2-3 min each, with real reports:
    • ubuntu-latest: found the go install module-path case mismatch (broken either way — tngtech 404s, TNG conflicts with go.mod's declared path), the AppArmor/bwrap gap on 24.04+, and a missing libsecret-1-0 in the Quickstart's apt install line.
    • macos-latest: found the Homebrew tap-trust gap again, plus a minor doctor/prose contradiction (built-in skill reported "missing" before the first omac start despite the README saying no separate step is needed).

The workflow is green and doing its intended job — real onboarding sessions, real README gaps surfaced, no false failures from the harness itself. Not merging without your go-ahead.

@nhuelstng nhuelstng merged commit 4bac617 into main Jul 15, 2026
7 checks passed
@nhuelstng nhuelstng deleted the fix/e2e-readme-onboarding-auto-timeout branch July 15, 2026 11:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant