test(e2e): run model-free e2e tests at PR time; assert --no-sandbox exposure#100
Merged
Conversation
…xposure The whole internal/e2e/ package sits behind //go:build e2e, and that tag is compiled only by e2e.yml (weekly cron + manual dispatch). ci.yml and release.yml run `go test ./...` without it, so the model-free security regression tests never ran on the merge path — including the ones explicitly built to run fast without a live model. A reintroduced dir_token leak (#74) or a silently-neutered assertion (#84) would pass CI and only surface the following Saturday. Add an `e2e_fast` build tag shared with the model-free files (`e2e || e2e_fast`) and a PR-time CI job that runs just that slice on ubuntu + macOS in ~3s: no harness install, no API token, no OS sandbox. The full live suite still runs weekly via e2e.yml (those files build under both tags). Also close the --no-sandbox gap (#66): runSecurityAudit no longer silently skips every negative assertion for a --no-sandbox harness. It now asserts the audit ran to completion and documents the per-property exposure surface, so a no-sandbox run that produced no audit fails loudly instead of passing green with nothing checked. The classification logic is a pure, unit-tested function (noSandboxExposureReport) that runs in the fast PR slice. Refactor: extract the pure assertion predicates (fs*Leaked, fsAllowDenied, secretLeaked, envVarsLeaked, netProbeDenied) into security_assertions.go and buildOmac into fixtures.go so both build modes share them; the *testing.T assert wrappers stay in e2e_test.go. Verification: - go test -tags=e2e_fast -race ./internal/e2e/ → 11 tests pass (~3s) - go vet -tags=e2e ./internal/e2e/ → full live suite still compiles - go build ./... && go vet ./... → untagged build unaffected Refs #66 Signed-off-by: Niclas Hülsmann <niclas.huelsmann@tngtech.com>
4 tasks
Contributor
Author
|
Follow-up release-procedure gap filed as #101 (release currently gates only on the untagged |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
e2e_fastbuild tag on the model-free files ininternal/e2e/(//go:build e2e || e2e_fast) and a new E2E fast CI job that runs just that slice onubuntu-latest+macos-lateston every push/PR.--no-sandboxgap:runSecurityAuditno longer silently skips all negative assertions — it asserts the audit ran to completion and documents the per-property exposure surface.buildOmacinto shared, fast-buildable files.Why
The entire
internal/e2e/package sits behind//go:build e2e, and that tag is compiled only bye2e.yml(weekly cron + manual dispatch).ci.ymlandrelease.ymlrungo test ./...without it, so the model-free security-regression tests never ran on the merge path — even the ones explicitly built to run fast without a live model:TestE2EServeDirTokenIsolationsecurity_assertions_test.goversions_test.go,harnesses_test.goConsequence: a reintroduced dir_token leak or a neutered fs-deny assertion would pass CI, pass the release gate, and ship — surfacing up to a week later. This moves the fast net onto the code path that produces releases.
For
--no-sandboxharnesses (codex/macOS), the audit previously skipped every negative assertion, silently excluding the real risk surface (#66). It now fails loudly if the audit produced nothing and logs an explicit EXPOSED/contained table per property.How
e2e || e2e_fastonharnesses.go,versions.go,classify.go,fixtures.go(new),security_assertions.go(new), and the four model-free test files. The live files (e2e_test.go,allowance.go,artifacts.go,provenance_test.go) staye2e-only.e2e.yml's-tags=e2estill runs everything weekly.noSandboxExposureReportis a pure function unit-tested in the fast slice (TestNoSandboxExposureReport), in the repo's "mutation-test the tests" spirit.omac serve --no-inneris control-plane only.Verification
Follow-up
A separate release-procedure gap (release currently gates only on untagged
go test) will be filed as its own issue.Refs #66
🤖 Generated with Claude Code