Cycle 8: bind the grade to the witnessed ledger + external-verdict honesty#15
Merged
Conversation
Forum's massively important role is the verifiable causal ledger; this extends it to the whole market. import-trace folds an external agent trace (LangSmith / OTel / AgentOps / generic JSON) into forum's hash-chained, replayable ledger, so ANY framework's run gains tamper-evidence (verify), causal replay, and a Merkle root — a flight recorder whose recording refutes its own tampering, which no observability tool (LangSmith, Langfuse, Arize, AgentOps) can do: their traces are mutable logs you must trust. Causal parents resolve by id -> ledger seq; an unknown parent is a root with the dangling id kept (counted, never guessed). FAIL LOUD: a malformed event raises rather than silently dropping a step. Zero-dep, deterministic. CLI: forum import-trace --format. 8 falsifiers: trace verifies + deterministic root, causal parents resolved, tampering breaks verification, 3 formats normalize, langsmith end-to-end, unknown-parent-is-root-not-guess, fail-loud, CLI. On feat/flight-recorder (off main, not disrupting the release branch). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…p unused merkle_root import and seqs local) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…str] (mypy) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…wheel bridge) Turns a finished orchestration run — native, or one folded in from any framework by the flight recorder — into a forum.gradable-trajectory/1 datum a training loop can admit: a prompt, the trajectory, and a GRADE that can fail. - grade.py: deterministic outcome grade over the ledger. Counts only independent checks (a producing actor cannot grade its own result) and returns UNVERIFIABLE when nothing independent checked the run — integrity is never laundered into success. - gradable_export.py: emits the sealed record with the merkle root + grade inputs a consumer re-derives OFF-forum with stdlib sha256 alone. Deliberately does NOT stamp a top-level 'witnessed' — forum reports only its own deep-verify; the witness is earned on the consumer side, never asserted here. - flight_recorder: extract ledger_from_trace so the pipeline composes. - cli: 'forum grade', 'forum export-gradable', and one-command 'forum mine' (trace -> verifiable ledger -> witnessed gradable RL data). 21 falsifiers incl. flipped-verdict-flips-grade, self-check-is-unverifiable, zero-checks-is-unverifiable, and a pinned cross-side merkle vector so the consumer's independent hash recipe cannot drift silently. 552 tests green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
One belief across every tool, consolidated: CREDO.md mirrors the canonical content-addressed text the Flywheel engine serves at GET /api/credo; the README carries the short form. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…dger
The gradable export was exactly the self-vouching theatre it exists to
avoid: grade_inputs carried a bare {actor, ok, kind} with no reference to
any witnessed entry, the merkle root covered only entry hashes, and the
check payload BODIES (where the ok truth values live) were never exported.
So an adversary could flip a grade_input.ok false->true, inflate reward to
1.0 / label PASS, recompute row_hash, leave the entries untouched, and the
consumer's promised re-derivation ('recompute reward from grade_inputs')
still matched because it recomputed over forum's own asserted numbers. The
PASS witnessed itself.
Each grade input now binds to its witnessed entry (seq + payload_hash), and
the export carries the check payload bodies, each of which hashes to a
witnessed entry's payload_hash (payload_hash -> entry_hash -> merkle_root).
A consumer reads ok from the merkle-bound body, not the free-floating
grade_input, and recomputes reward from the bound checks. Flipping a
grade_input no longer changes the derived grade; flipping a check body
breaks its hash binding. The grade cannot be forged apart from the
witnessed record. The pinned cross-side merkle vector is unchanged.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…atus in the brief The cycle-8 grade-binding work made the run summary ledger-scoped (an intact ledger is not answer acceptance) but left two tests red: the honesty test wanted an external- verification BULLET that was never added, and a run-room golden still asserted the old summary. _brief_bullets now emits an honest external-verification line: "none ran (an intact ledger is not answer acceptance)" when no external check ran, else the ran/refuted counts. A "complete" brief can no longer read as if the answer was accepted from the ledger alone. The stale golden is updated to the intended honest summary + bullet. 563 passed, 2 skipped (was 2 failed). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…code) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…rederived The portable receipt now names the EXTERNAL verifier's ruling on the answer (answer_verified) distinct from hash-chain integrity: a refutation never reads MATCH, an abstention (forum standing alone by design) surfaces the honest null rather than omitting the external ruling. The roster rejects duplicate agent names (a duplicate double-counts an actor and lets a second row impersonate the first). run_room signals separate 'a verification actually ran' from 'the ledger re-derived', and semantic intent-drift from the lexical check. 563 tests pass, 2 new falsifier files. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Cycle-8 honesty pass across forum's judgment surface, plus the second-person README register sweep.
563 tests pass, 2 skipped.
🤖 Generated with Claude Code