Flight recorder + gradable export: any framework's trace → verifiable ledger → witnessed RL data#13
Merged
Merged
Conversation
Forum's massively important role is the verifiable causal ledger; this extends it to the whole market. import-trace folds an external agent trace (LangSmith / OTel / AgentOps / generic JSON) into forum's hash-chained, replayable ledger, so ANY framework's run gains tamper-evidence (verify), causal replay, and a Merkle root — a flight recorder whose recording refutes its own tampering, which no observability tool (LangSmith, Langfuse, Arize, AgentOps) can do: their traces are mutable logs you must trust. Causal parents resolve by id -> ledger seq; an unknown parent is a root with the dangling id kept (counted, never guessed). FAIL LOUD: a malformed event raises rather than silently dropping a step. Zero-dep, deterministic. CLI: forum import-trace --format. 8 falsifiers: trace verifies + deterministic root, causal parents resolved, tampering breaks verification, 3 formats normalize, langsmith end-to-end, unknown-parent-is-root-not-guess, fail-loud, CLI. On feat/flight-recorder (off main, not disrupting the release branch). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…p unused merkle_root import and seqs local) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…str] (mypy) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…wheel bridge) Turns a finished orchestration run — native, or one folded in from any framework by the flight recorder — into a forum.gradable-trajectory/1 datum a training loop can admit: a prompt, the trajectory, and a GRADE that can fail. - grade.py: deterministic outcome grade over the ledger. Counts only independent checks (a producing actor cannot grade its own result) and returns UNVERIFIABLE when nothing independent checked the run — integrity is never laundered into success. - gradable_export.py: emits the sealed record with the merkle root + grade inputs a consumer re-derives OFF-forum with stdlib sha256 alone. Deliberately does NOT stamp a top-level 'witnessed' — forum reports only its own deep-verify; the witness is earned on the consumer side, never asserted here. - flight_recorder: extract ledger_from_trace so the pipeline composes. - cli: 'forum grade', 'forum export-gradable', and one-command 'forum mine' (trace -> verifiable ledger -> witnessed gradable RL data). 21 falsifiers incl. flipped-verdict-flips-grade, self-check-is-unverifiable, zero-checks-is-unverifiable, and a pinned cross-side merkle vector so the consumer's independent hash recipe cannot drift silently. 552 tests green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
HarperZ9
changed the title
One belief across every tool, consolidated: CREDO.md mirrors the canonical content-addressed text the Flywheel engine serves at GET /api/credo; the README carries the short form. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Turns forum into the manufacturing line that converts any framework's agent run into witnessed, gradable RL training data — the data the field says it is running out of (Depue, A Stargate for Data), manufactured with a receipt scraped tokens lack.
Two composable halves:
1. Flight recorder (
flight_recorder.py,import-trace) — folds a LangSmith / OpenTelemetry / AgentOps / generic JSON trace into forum's hash-chained, replayable ledger with a Merkle root. Tamper-evidence no observability tool has: the recording refutes its own tampering.2. Gradable export (
grade.py,gradable_export.py,export-gradable+ one-commandmine) — turns a finished run's ledger into aforum.gradable-trajectory/1datum:UNVERIFIABLEwhen nothing independent checked the run. Integrity is never laundered into success.witnessed:true. The witness is earned on the consumer side by re-derivation, never asserted by the exporter.forum mine <trace> -o data.jsonldoes the whole pipeline in one command.Why it beats the class: LangSmith / Langfuse / AgentOps / OTel all export a mutable log you must trust, with no success signal and no re-checkable proof. No competitor exports a graded, tamper-evident, independently re-checkable trajectory a training loop can admit.
Tests: 29 falsifiers total (8 flight recorder + 6 grade + 6 export + 1 mine CLI + 8 cross-repo consumer). Includes a pinned merkle vector an independent stdlib re-implementation reproduces exactly, so the two ends' hash recipes cannot drift silently. Full suite 552 passed, ruff + mypy clean.
🤖 Generated with Claude Code