Skip to content

test(e2e): run model-free e2e tests at PR time; assert --no-sandbox exposure#100

Merged
nhuelstng merged 1 commit into
mainfrom
e2e-fast-pr-gate
Jul 15, 2026
Merged

test(e2e): run model-free e2e tests at PR time; assert --no-sandbox exposure#100
nhuelstng merged 1 commit into
mainfrom
e2e-fast-pr-gate

Conversation

@nhuelstng

Copy link
Copy Markdown
Contributor

What

  • Add an e2e_fast build tag on the model-free files in internal/e2e/ (//go:build e2e || e2e_fast) and a new E2E fast CI job that runs just that slice on ubuntu-latest + macos-latest on every push/PR.
  • Close the --no-sandbox gap: runSecurityAudit no longer silently skips all negative assertions — it asserts the audit ran to completion and documents the per-property exposure surface.
  • Refactor the pure assertion predicates and buildOmac into shared, fast-buildable files.

Why

The entire internal/e2e/ package sits behind //go:build e2e, and that tag is compiled only by e2e.yml (weekly cron + manual dispatch). ci.yml and release.yml run go test ./... without it, so the model-free security-regression tests never ran on the merge path — even the ones explicitly built to run fast without a live model:

Test Guards
TestE2EServeDirTokenIsolation the real #74 dir_token cross-workdir leak, over real loopback sockets
security_assertions_test.go assertions silently becoming no-ops (the #84 rigor gap / #66 "mutation-test the tests")
versions_test.go, harnesses_test.go harness/version registry invariants

Consequence: a reintroduced dir_token leak or a neutered fs-deny assertion would pass CI, pass the release gate, and ship — surfacing up to a week later. This moves the fast net onto the code path that produces releases.

For --no-sandbox harnesses (codex/macOS), the audit previously skipped every negative assertion, silently excluding the real risk surface (#66). It now fails loudly if the audit produced nothing and logs an explicit EXPOSED/contained table per property.

How

  • e2e || e2e_fast on harnesses.go, versions.go, classify.go, fixtures.go (new), security_assertions.go (new), and the four model-free test files. The live files (e2e_test.go, allowance.go, artifacts.go, provenance_test.go) stay e2e-only.
  • Full live suite is unchanged: those files build under both tags, so e2e.yml's -tags=e2e still runs everything weekly.
  • noSandboxExposureReport is a pure function unit-tested in the fast slice (TestNoSandboxExposureReport), in the repo's "mutation-test the tests" spirit.
  • No bubblewrap / token / harness install in the fast job — omac serve --no-inner is control-plane only.

Verification

go test -tags=e2e_fast -race ./internal/e2e/   # 11 tests pass (~3s), incl. real serve subprocess
go vet  -tags=e2e       ./internal/e2e/         # full live suite still compiles
go build ./... && go vet ./...                  # untagged build unaffected

Follow-up

A separate release-procedure gap (release currently gates only on untagged go test) will be filed as its own issue.

Refs #66

🤖 Generated with Claude Code

…xposure

The whole internal/e2e/ package sits behind //go:build e2e, and that tag is
compiled only by e2e.yml (weekly cron + manual dispatch). ci.yml and
release.yml run `go test ./...` without it, so the model-free security
regression tests never ran on the merge path — including the ones explicitly
built to run fast without a live model. A reintroduced dir_token leak (#74) or
a silently-neutered assertion (#84) would pass CI and only surface the
following Saturday.

Add an `e2e_fast` build tag shared with the model-free files (`e2e || e2e_fast`)
and a PR-time CI job that runs just that slice on ubuntu + macOS in ~3s: no
harness install, no API token, no OS sandbox. The full live suite still runs
weekly via e2e.yml (those files build under both tags).

Also close the --no-sandbox gap (#66): runSecurityAudit no longer silently
skips every negative assertion for a --no-sandbox harness. It now asserts the
audit ran to completion and documents the per-property exposure surface, so a
no-sandbox run that produced no audit fails loudly instead of passing green
with nothing checked. The classification logic is a pure, unit-tested function
(noSandboxExposureReport) that runs in the fast PR slice.

Refactor: extract the pure assertion predicates (fs*Leaked, fsAllowDenied,
secretLeaked, envVarsLeaked, netProbeDenied) into security_assertions.go and
buildOmac into fixtures.go so both build modes share them; the *testing.T
assert wrappers stay in e2e_test.go.

Verification:
- go test -tags=e2e_fast -race ./internal/e2e/  → 11 tests pass (~3s)
- go vet -tags=e2e ./internal/e2e/              → full live suite still compiles
- go build ./... && go vet ./...                → untagged build unaffected

Refs #66

Signed-off-by: Niclas Hülsmann <niclas.huelsmann@tngtech.com>
@nhuelstng

Copy link
Copy Markdown
Contributor Author

Follow-up release-procedure gap filed as #101 (release currently gates only on the untagged go test; the e2e_fast slice added here doesn't yet run before a tag is cut).

@nhuelstng
nhuelstng merged commit ddd3b38 into main Jul 15, 2026
7 checks passed
@nhuelstng
nhuelstng deleted the e2e-fast-pr-gate branch July 15, 2026 12:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant