Test that a blocked action leaves no trace of having happened - #104
Merged
b-macker merged 3 commits intoJul 29, 2026
Conversation
The governance engine already gates actions before they execute, and already writes a signed RefusalAttestation when it does. What was missing was a test tying the two halves together: that the side effect never formed, AND that the refusal can be proven afterward from evidence an attacker cannot quietly edit. Exit codes cannot carry that claim. A run blocked at the boundary and a run that never started both exit non-zero and leave no file behind, so asserting absence proves nothing on its own. Group A runs the same polyglot block under a permissive policy and confirms it DOES write its sentinel; Group B then flips only the policy and confirms it does not. The control is what gives the absence meaning — verified by inverting the Group B policy, which turns B-02 into a failure rather than a silent pass. Groups C and D cover the proof half: the attestation is Ed25519-signed, names the rule that refused, and is chained — flipping execution_prevented from true to false is reported as TAMPER, and deleting the record breaks the chain. Group E repeats the whole shape on the shell capability gate to show the guarantee is not specific to the language check. The existing naab-41 P2 check accepts exit 3 alone as proof of non-formation, so it passes even if no attestation is written. This does not. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ELUfjXZvx8kzXo1UJjrAhC
NAAb Governance Report
All governance checks passed! Generated by NAAb Governance Engine v4.0 |
The Windows job runs a native binary under an MSYS2 harness. Absolute POSIX paths written into govern.json are not paths that binary can open, so the telemetry file was never created: fopen returned NULL and emitRefusalAttestation returned early without writing anything. The failure shape is worth noting. Groups A, B and E-01/E-02 passed on Windows — the action really was blocked and really did leave no side effect. Only the evidence went missing, which is exactly the case where a test that asserted exit codes alone would have reported success. Fixed the way test_evidence_chain.sh already does it: relative output_file and signing_key in govern.json, with every invocation, the chain verifier included, run from inside the test directory. The signing key is copied in alongside so it resolves the same way. Also verified the jq-absent branch of refusal_event, which is the branch CI actually takes and which the local run had been skipping. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ELUfjXZvx8kzXo1UJjrAhC
b-macker
marked this pull request as ready for review
July 29, 2026 00:03
…sible-action-prevention-4cmn1m
9 tasks
b-macker
added a commit
that referenced
this pull request
Jul 30, 2026
…106) living-script_extended demonstrated 23 governance levels and produced zero signed attestations: telemetry.tamper_evidence was on with no audit section at all. It also built a tamper-evident chain nothing ever verified, and archived telemetry and transcript but no audit file — two more instances of the "configured but unobserved" gap its own run.sh header records as the reason levels 19b/21/22/23 exist. Validating the config against the stub before writing the level surfaced a real defect. ed25519Fingerprint() reads with PEM_read_bio_PUBKEY and refuses private keys by design, but all three emit*Attestation sites handed it the signing key, so fingerprinting silently returned "" into the surrounding catch. Every signed attestation NAAb had written carried a signature that could not be attributed to a key. Fixed by fingerprinting the derived public half via the existing ed25519PublicFromPrivate(); the value now equals what the trust store reports for the same key. The C-05 assertion merged in #104 could not have caught this — it grepped '"key_fingerprint":"', which also matches an empty value — so it now requires hex content. The two attestation kinds go to different files behind different gates: refusal attestations to telemetry, execution attestations through logAuditEvent to the audit file, gated additionally on audit.level != "none" which defaults to none. Refusal attestations only exist when governance blocked something, so the level asserts agreement with their count rather than a floor. The live keyed run corrected L24-02. It first asserted one attestation per script-counted send and failed at 22 vs 21; adding commits to the denominator would not have fixed it either, since 21 + 2 is 23. Exact equality is unsound here because agent.commit() also attests, safe_send() increments the counter even when a send was blocked before reaching emitAttestation, and several send-like APIs bump it with their own semantics. L24-02 now asserts only what is behaviour-independent: attestations exist, every one carries a known action, and attested sends cannot outnumber attempted sends. Live run confirmed the rest: 22 signed with 0 empty fingerprints, 43 snapshots = 21 semantic + 22 OA, both chains verified, and the audit section survived 15 accepted operator rewrites with no ratchet rejection touching provenance. Full suite: 441 tests, 0 unexpected failures. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
5 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds an end-to-end test for the question "can an inadmissible action be prevented from becoming operationally real, and can that prevention be proven afterward?"
Both halves already worked — the engine gates actions before they execute and writes a signed
RefusalAttestationwhen it refuses. What was missing was a test tying them together and measuring prevention as an observable side effect rather than an exit code.Exit codes cannot carry the claim. A run blocked at the boundary and a run that never started both exit non-zero and leave no file behind, so asserting absence proves nothing on its own. The test therefore runs the same polyglot block twice, changing only the policy.
Changes
tests/governance_v4/test_nonformation_proof.sh(15 checks, no API key needed):languages.blockedpolicy: exit 3, sentinel never appears, execution does not continue past the boundaryexecution_prevented:true,binding_status:non-binding, the refusing rule named, Ed25519 signature + key fingerprint presentexecution_preventedtofalseis reported asTAMPER, deleting the record breaks the hash chainrun-all-tests.shalongside the other governance_v4 shell teststests/gorilla/naab-41/run-naab41.shP2 already covers "proof of non-formation", but accepts exit 3 alone as sufficient, so it passes even when no attestation is written. This test does not.Test Plan
bash tests/governance_v4/test_nonformation_proof.sh— 15/15 passside effect file exists — the blocked action became realrather than a silent pass, and takes C-01..C-05 with ittest_tool_admissibility_gate.sh6/6,test_split_commit.sh11/11,test_uncatchable.sh14/14,naab-4166/66 (23/23 deterministic)bash run-all-tests.sh— not run end-to-end here; the runner edit is a copy of the existing registration block and passesbash -nNo source files changed, so no error-message text was affected.
Related Issues
None.
Generated by Claude Code