Skip to content

Attribute attestation signatures, and prove the evidence layer fires - #106

Draft
b-macker wants to merge 2 commits into
masterfrom
claude/naab-inadmissible-action-prevention-4cmn1m
Draft

Attribute attestation signatures, and prove the evidence layer fires#106
b-macker wants to merge 2 commits into
masterfrom
claude/naab-inadmissible-action-prevention-4cmn1m

Conversation

@b-macker

Copy link
Copy Markdown
Owner

Summary

living-script_extended demonstrated 23 governance levels and produced zero signed attestationstelemetry.tamper_evidence was on with no audit section at all. It also built a tamper-evident chain nothing ever verified, and archived telemetry and transcript but no audit file. Two more instances of the "configured but unobserved" gap its own run.sh header records as the reason levels 19b/21/22/23 exist.

The defect this surfaced

Validating the config against the stub before writing the level found a real bug:

ed25519Fingerprint() reads with PEM_read_bio_PUBKEY and refuses private keys by design, but all three emit*Attestation sites handed it the signing key. Fingerprinting silently returned "" into the surrounding catch (...) {}. Every signed attestation NAAb has ever written carries a signature that cannot be attributed to a key.

Fixed by fingerprinting the derived public half via the existing ed25519PublicFromPrivate(), matching how governance_engine.cpp:4123 already does it. The fingerprint now equals what the trust store reports for the same key:

attestation key_fingerprint: c223415f60114f4d8f861d8be037e59f
naab-lang --list-keys:       c223415f60114f4d8f861d8be037e59f

The C-05 assertion merged in #104 could not have caught this — it greps '"key_fingerprint":"', which also matches "". Tightened to require hex content. The new level asserts fingerprint content for the same reason.

Why the level is shaped this way

The two attestation kinds go to different files behind different gates:

Kind Destination Gate
Refusal telemetry provenance.enabled + record_attestations
Execution audit above + audit.level != "none" (defaults to none)

Refusal attestations only exist when governance actually blocked something, so asserting a floor on them would demand the agent misbehave. Execution attestations fire once per agent.send(), so they're the countable population. Refusals are checked for agreement with their attestations instead — zero of both is a pass.

That's also why audit.level and archiving the audit file are required, not optional: without them the level has nothing deterministic to assert.

Changes

  • src/runtime/governance_reports.cpp — fingerprint the derived public key at all 3 attestation sites. trust_store.cpp untouched; it already receives a public PEM.
  • tests/governance_v4/test_nonformation_proof.sh — C-05 requires [0-9a-f]{16,}, not mere presence.
  • src/govern.jsonaudit section (level: basic, tamper_evidence, provenance with signing_key_env, not a literal path — the key lives in an ephemeral $TEST_TMP) plus telemetry.decision_snapshots.
  • src/living-script.naab — new EVIDENCE_AUDIT phase. Counting stays in grep/wc because decision_snapshots pushes telemetry past a megabyte and it must not enter script memory. Added a -F helper: process.run passes argv with no shell, so grep sees one backslash and its BRE reads \" as a plain ", which never matches the escaped attestation body.
  • run.sh — Level 24 assertions, archive audit.jsonl, and verify both chains.

Test Plan

  • bash run-all-tests.sh441 tests, 0 unexpected failures

  • test_nonformation_proof.sh 15/15 with the tightened C-05

  • tests/security/test_error_msg_leaks.sh — 874/0 (attestation code touched)

  • End-to-end without live keys, via tests/helpers/agent_stub.py: 3 sends → 3 execution attestations, all signed with non-empty fingerprints, cdd_snapshot on every SEMANTIC_TURN and OUTPUT_ADMISSIBILITY_EVAL (6 = 3+3), both chains Chain verified:

  • The real phase code, extracted verbatim from the script, run against those real files — correct markers

  • Every L24 assertion checked against its own degraded case; each fails only there:

    scenario L24-02 L24-03 L24-04
    healthy PASS PASS PASS
    pre-fix engine (empty fingerprints) PASS FAIL PASS
    record_attestations off FAIL PASS PASS
    decision_snapshots off PASS PASS FAIL
    partial snapshot coverage PASS PASS FAIL
  • parse clean; check diagnostics unchanged at 35; run.sh passes bash -n; govern.json valid JSON

  • Live keyed run — not possible here. Two things to confirm on the first real run: that the audit section survives the operator's catjson.parsejson.stringify → re-sign round-trip (if dropped, record_attestations goes true→false and the evidence ratchet rejects every later reload), and that no CONFIG_ADJUSTMENT carries accepted:false with reason:ratchet.

Notes

L24-03 reads PASS when there are zero attestations (0-of-0 signed), but L24-02 catches that case, so the pair is complete.

Left deliberately out of scope: the pre-existing NAAB_SIGN_PATH / NAAB_SIGNING_KEY naming mismatch; and taint_tracking / temporal_coupling / integrity, each of which needs a deliberate observable sink to avoid being inert.


Generated by Claude Code

living-script_extended demonstrated 23 governance levels and produced zero
signed attestations: it had telemetry.tamper_evidence on and no audit section
at all. It also built a tamper-evident chain that nothing ever verified, and
archived telemetry and transcript but no audit file — two more instances of
the "configured but unobserved" gap its own header records as the reason
levels 19b/21/22/23 exist.

Validating the config against the stub before writing any of it surfaced a
real defect. ed25519Fingerprint() reads with PEM_read_bio_PUBKEY and refuses
private keys by design, but all three emit*Attestation sites handed it the
signing key, so fingerprinting silently returned "" into the surrounding
catch. Every signed attestation NAAb has written carries a signature that
cannot be attributed to a key. Fixed by fingerprinting the derived public
half via the existing ed25519PublicFromPrivate(), matching how
governance_engine.cpp:4123 already does it; the fingerprint now equals the
one the trust store reports for the same key.

The C-05 assertion merged in #104 could not have caught this: it greps
'"key_fingerprint":"' which also matches an empty value. It now requires hex
content. The new level asserts fingerprint content for the same reason.

Two attestation kinds go to different files behind different gates. Refusal
attestations land in telemetry; execution attestations go through
logAuditEvent to the audit file, gated additionally on audit.level != "none"
which defaults to none. That distinction decides the level's design: refusal
attestations only exist when governance blocked something, so asserting a
floor on them would demand the agent misbehave. Execution attestations fire
once per send, so they are the countable population. Refusals are therefore
checked for AGREEMENT with their attestations — zero of both passes.

Counting stays in grep/wc because decision_snapshots pushes telemetry past a
megabyte and it must not enter script memory. The audit patterns need -F:
process.run passes argv with no shell, so grep sees one backslash and its BRE
reads \" as a plain ", which never matches the escaped attestation body.

Verified without live keys: 3 sends produce 3 attestations, all signed with a
non-empty fingerprint, snapshots on every SEMANTIC_TURN and
OUTPUT_ADMISSIBILITY_EVAL, both chains verifying. Each L24 assertion was
checked against its own degraded case and fails only there.

Full suite: 441 tests, 0 unexpected failures.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ELUfjXZvx8kzXo1UJjrAhC
@github-actions

Copy link
Copy Markdown

NAAb Governance Report

Metric Count
Files checked 16
Passed 16
Failed 0

All governance checks passed!

Generated by NAAb Governance Engine v4.0

…count

The live keyed run reported L24-02 failing: 22 execution attestations against
21 script-counted sends. The denominator was wrong, but adding commits to it
would not have fixed the check — 21 + 2 commits is 23, not 22.

Exact equality is not a sound invariant here, for three independent reasons:

  - agent.commit() attests as well as agent.send() (agent_impl.cpp:5382), so
    total_sends was never the right denominator.
  - safe_send() catches a throw but the script still increments total_sends, so
    a send blocked before reaching emitAttestation counts as a send and
    produces no attestation. That makes equality swing on agent behaviour.
  - agent.run/batch/fan_out/pipeline each bump the counter by one or two, and
    whether each attests is a separate question per API.

The second reason is exactly why L24-05 asserts agreement rather than a floor
for refusal attestations. I reasoned it out for that check and then wrote the
neighbouring one as a hard equality anyway.

L24-02 now asserts what holds regardless of behaviour: attestations exist,
every one carries a known action (send + commit must sum to the total, so an
unaccounted action type fails), and attested sends cannot outnumber attempted
sends (a phantom attestation fails). The script reports the send/commit split
and the script-side count for forensics instead of asserting on them.

Verified against the live-reported numbers: 22 = 20 send + 2 commit with 21
attempted now passes, while record_attestations off, an unaccounted action
type, and phantom attestations each still fail.

The rest of the live run confirmed the change: 22 signed with 0 empty
fingerprints (the fix in this PR), 43 snapshots = 21 semantic + 22 OA, both
chains verified, the audit section survived 15 accepted operator rewrites with
no ratchet rejection touching provenance, and telemetry grew 550KB -> 1.3MB.

Full suite: 441 tests, 0 unexpected failures.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ELUfjXZvx8kzXo1UJjrAhC
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants