Skip to content

fix(e2e): read audit-output.txt in TestE2EProvenance instead of chat text#60

Merged
nhuelstng merged 1 commit into
mainfrom
fix/provenance-audit-output-file
Jul 13, 2026
Merged

fix(e2e): read audit-output.txt in TestE2EProvenance instead of chat text#60
nhuelstng merged 1 commit into
mainfrom
fix/provenance-audit-output-file

Conversation

@nhuelstng

Copy link
Copy Markdown
Contributor

What

TestE2EProvenance now reads audit-output.txt from disk (with a fallback to agent stdout/stderr) instead of asserting solely on the agent's chat reply.

Why

The latest scheduled E2E run on main (run 29146467693) failed TestE2EProvenance on 2 of 7 matrix jobs (opencode/macos-latest, copilot/ubuntu-latest) with:

behavior mismatch: "blocked.example.com" not denied by sandbox (audit output lacks denial message)
behavior mismatch: no filesystem denial in audit output despite provenance listing protected paths as denied

Downloading the uploaded session artifacts showed the sandbox actually enforced everything correctly — DENY blocked.example.com, Permission denied on /etc/shadow, etc. all appear in audit-output.txt and the omac logs for the failing runs. The test was reading the wrong source: self-audit/scripts/audit.sh redirects all its output to a file via exec > "$_audit_file" 2>&1 (some harness TUIs collapse tool output), and TestE2EProvenance asserted only on the agent's free-form chat reply, which only contains the denial markers when the model happens to cat the file back verbatim. TestE2ESecurityAudit already reads audit-output.txt directly for exactly this reason — TestE2EProvenance never got the same treatment.

How

Mirrors the existing pattern in e2e_test.go's TestE2ESecurityAudit: read workdir/audit-output.txt after runAuditAgent, combine with agent stdout, and fall back to agent-only output if the file is missing (e.g. the agent refused to run the script).

Verification

  • go build ./... and go build -tags=e2e ./... — clean
  • go vet -tags=e2e ./internal/e2e/... — clean
  • gofmt -l — clean
  • go test ./... (non-e2e suite) — all pass
  • Did not run the live -tags=e2e suite locally (requires bubblewrap, live harness installs, and CI secrets); the actual fix will be exercised by the next scheduled/dispatched E2E run in CI.

…text

audit.sh redirects all probe output to a file (some harness TUIs collapse
tool output), so asserting on the agent's free-form chat reply only
passes when the model happens to paste the file back verbatim. This
caused TestE2EProvenance to flake on main (e.g. run 29146467693): the
sandbox correctly denied network/filesystem access in every case, but
the test read the wrong source. TestE2ESecurityAudit already reads the
file directly; apply the same pattern here.

Signed-off-by: Niclas Hülsmann <niclas.huelsmann@tngtech.com>
@nhuelstng nhuelstng merged commit 703b63f into main Jul 13, 2026
5 of 11 checks passed
@nhuelstng nhuelstng deleted the fix/provenance-audit-output-file branch July 13, 2026 11:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant