fix(e2e): read audit-output.txt in TestE2EProvenance instead of chat text#60
Merged
Merged
Conversation
…text audit.sh redirects all probe output to a file (some harness TUIs collapse tool output), so asserting on the agent's free-form chat reply only passes when the model happens to paste the file back verbatim. This caused TestE2EProvenance to flake on main (e.g. run 29146467693): the sandbox correctly denied network/filesystem access in every case, but the test read the wrong source. TestE2ESecurityAudit already reads the file directly; apply the same pattern here. Signed-off-by: Niclas Hülsmann <niclas.huelsmann@tngtech.com>
This was referenced Jul 13, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
TestE2EProvenancenow readsaudit-output.txtfrom disk (with a fallback to agent stdout/stderr) instead of asserting solely on the agent's chat reply.Why
The latest scheduled E2E run on
main(run 29146467693) failedTestE2EProvenanceon 2 of 7 matrix jobs (opencode/macos-latest,copilot/ubuntu-latest) with:Downloading the uploaded session artifacts showed the sandbox actually enforced everything correctly —
DENY blocked.example.com,Permission deniedon/etc/shadow, etc. all appear inaudit-output.txtand the omac logs for the failing runs. The test was reading the wrong source:self-audit/scripts/audit.shredirects all its output to a file viaexec > "$_audit_file" 2>&1(some harness TUIs collapse tool output), andTestE2EProvenanceasserted only on the agent's free-form chat reply, which only contains the denial markers when the model happens tocatthe file back verbatim.TestE2ESecurityAuditalready readsaudit-output.txtdirectly for exactly this reason —TestE2EProvenancenever got the same treatment.How
Mirrors the existing pattern in
e2e_test.go'sTestE2ESecurityAudit: readworkdir/audit-output.txtafterrunAuditAgent, combine with agent stdout, and fall back to agent-only output if the file is missing (e.g. the agent refused to run the script).Verification
go build ./...andgo build -tags=e2e ./...— cleango vet -tags=e2e ./internal/e2e/...— cleangofmt -l— cleango test ./...(non-e2e suite) — all pass-tags=e2esuite locally (requires bubblewrap, live harness installs, and CI secrets); the actual fix will be exercised by the next scheduled/dispatched E2E run in CI.