diff --git a/CHANGELOG.md b/CHANGELOG.md index 0826115..d904b07 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,13 @@ # Changelog +## Unreleased + +- Deep verify benchmark: adds `forum bench-deep-verify`, a zero-dependency benchmark + for ledger integrity scaling. It times chain-only `verify()`, payload-only + `verify_payloads()`, and full `verify(deep=True)` across entry counts, payload body + sizes, storage modes, fsync behavior, redaction ratios, warmups, and repeats, with a + `forum.deep-verify-benchmark/v1` JSON receipt for public comparison. + ## 1.13.0 (2026-07-07): campaign orchestration, approval gates, and operator surfaces A run is no longer the largest unit Forum can witness. This release adds campaign orchestration above the run, puts a human approval gate (with a durable deadline) inside the run, and gives the operator live surfaces to watch and steer both: run rooms, runtime inspection, context preflight, and context capsules, all read from the same witnessed ledger. diff --git a/CREDO.md b/CREDO.md new file mode 100644 index 0000000..8e186bc --- /dev/null +++ b/CREDO.md @@ -0,0 +1,23 @@ +# The Credo + +One belief, held steady across every tool in the family. The canonical, +content-addressed copy is served by the Flywheel engine at `GET /api/credo`; +this file echoes it. + +Flywheel and its tool family hold one belief steady across every surface: + +1. Knowledge is an open surface for anyone who can attain the means; + we build to lower the means. +2. The work speaks for itself: an external check decides acceptance; + never reputation, never the model. +3. Every result carries a receipt a stranger can re-run. +4. No claim without its interval; the honest null is a first-class result. +5. Ownership is earned by comprehension: you own what you can review, + explain, and defend. +6. Learning is woven into the work: every flagged gap is a lesson about + your own system. +7. The badge and the contribution were bundled; we rebundle them around + the work, not the worker. + +Growth, learning, knowledge, application, ownership: in that order, +for everyone. \ No newline at end of file diff --git a/README.md b/README.md index 02e8d7b..04c2eb1 100644 --- a/README.md +++ b/README.md @@ -65,7 +65,7 @@ forum serve --chat-url http://localhost:11434/v1/chat/completions --model llama3 forum mcp --cmd "ollama run llama3" ``` -`forum --help` lists the full surface: `status`, `doctor`, `demo`, `humanize`, `route`, `submit`, `serve`, `mcp`, `context`, `runtime`, `ledger`, `gate`, `campaign`, `bench`. From a source checkout the same CLI is available as `python -m forum`. See [RUNNING.md](RUNNING.md) for real-model setups and [USAGE.md](USAGE.md) for the operator surface. +`forum --help` lists the full surface: `status`, `doctor`, `demo`, `humanize`, `route`, `submit`, `serve`, `mcp`, `context`, `runtime`, `ledger`, `gate`, `campaign`, `bench`, and `bench-deep-verify`. From a source checkout the same CLI is available as `python -m forum`. See [RUNNING.md](RUNNING.md) for real-model setups and [USAGE.md](USAGE.md) for the operator surface. ## A worked example: catch a tampered record @@ -100,6 +100,12 @@ Two old ideas do most of the work. A hash chain: every entry carries a fingerpri Everything else falls out of those two. `replay(until=...)` rebuilds the exact state at any past point. `causal_chain(seq)` follows parent links to answer why something happened. `checkpoint()` folds the history into one Merkle root, built to avoid the second-preimage collision (CVE-2012-2459) that naive Merkle code runs into. By default the ledger lives in memory; point it at `FileStorage` and every entry is appended to a JSONL file and fsynced before the next, so the record survives a restart, tolerates a crash-torn final write, and still verifies exactly. +To measure the scaling cost honestly, `forum bench-deep-verify` builds deterministic ledgers and times chain-only `verify()`, payload-only `verify_payloads()`, and full `verify(deep=True)` separately. It varies entry count, payload body bytes, storage mode (`memory`, `file-sync`, `file-batched`), and redaction ratio, then emits a `forum.deep-verify-benchmark/v1` JSON receipt: + +```bash +forum bench-deep-verify --entries 1000,10000 --payload-bytes 256,4096 --storage memory --storage file-batched --json +``` + ## HTTP and MCP surfaces The daemon exposes route, plan, submit, humanize, prose contracts, gates, run rooms, capsules, runtime inspection, context preflight, and ledger verify/replay over HTTP (`/route`, `/plan`, `/submit`, `/gates`, `/gate/approve`, `/room`, `/capsule`, `/runtime`, `/context/preflight`, `/prose/contract`, `/verify`, and more). MCP mirrors the same tools: `forum.submit`, `forum.route`, `forum.plan`, `forum.status`, `forum.doctor`, `forum.verify`, `forum.prose.humanize`, `forum.prose.contract`, `forum.ledger.summary`, `forum.ledger.capsule`, `forum.ledger.get`, `forum.run.room`, `forum.runtime.inspect`, `forum.context.preflight`, and `forum.gate.list` / `approve` / `edit` / `reject`. @@ -137,3 +143,12 @@ python -m pip install -e ".[dev]" python -m pytest python examples/demo.py ``` + +## What this believes + +This tool is one lane of a family that holds a single belief steady across +every surface: knowledge open to anyone who can attain the means; acceptance +decided by external checks, never reputation; every result re-runnable; +honest nulls first-class; ownership earned by comprehension; learning woven +into the work. The full text lives in [CREDO.md](CREDO.md). +The long form of this belief: [The Unbundling](https://github.com/HarperZ9/flywheel/blob/fix/release-model-identity/docs/essays/2026-07-13-the-unbundling.md). diff --git a/USAGE.md b/USAGE.md index 952fae8..47fc916 100644 --- a/USAGE.md +++ b/USAGE.md @@ -61,6 +61,22 @@ brief: latest request, latest answer, task results, quality signals, checkpoint, verification state. `--use-capsule-context` feeds that brief through the normal ContextProvider seam, so existing context budgets still decide how much is admitted. +## Deep Verify Benchmark + +```bash +forum bench-deep-verify --json +forum bench-deep-verify --entries 1000,10000 --payload-bytes 256,4096 --storage memory --storage file-batched --redaction-ratio 0,0.5,1 --json --out deep-verify.json +``` + +`bench-deep-verify` measures the scaling cost of the causal ledger's integrity checks. +It reports chain-only `verify()`, payload-only `verify_payloads()`, and full +`verify(deep=True)` timings as a `forum.deep-verify-benchmark/v1` receipt. The +variables are entry count, payload body bytes, storage mode, fsync mode, redaction +ratio, warmups, and repeats. Redacted payload bodies are removed before verification, +so the benchmark also shows the content-addressed trade-off: the chain can still +verify when only fingerprints remain, while deep payload rehashing scales with the +payload bodies that are still present. + ## Expert Delivery Profiles ```bash diff --git a/src/forum/bench_deep_verify.py b/src/forum/bench_deep_verify.py new file mode 100644 index 0000000..aa8d6c1 --- /dev/null +++ b/src/forum/bench_deep_verify.py @@ -0,0 +1,253 @@ +from __future__ import annotations + +import json +import platform +import statistics +import sys +import tempfile +import time +from collections.abc import Iterable +from dataclasses import dataclass +from typing import Any + +from forum.ledger import InMemoryStorage, Ledger, LedgerEntry, Storage +from forum.storage import FileStorage + +SCHEMA = "forum.deep-verify-benchmark/v1" +STORAGE_MODES = ("memory", "file-sync", "file-batched") + + +@dataclass(frozen=True, slots=True) +class BenchmarkCase: + entry_count: int + payload_body_bytes: int + storage_mode: str + redaction_ratio: float + + +def parse_int_csv(text: str) -> list[int]: + values: list[int] = [] + for raw in text.split(","): + raw = raw.strip() + if not raw: + continue + value = int(raw) + if value < 0: + raise ValueError("integer lists cannot contain negative values") + values.append(value) + if not values: + raise ValueError("expected at least one integer") + return values + + +def parse_float_csv(text: str) -> list[float]: + values: list[float] = [] + for raw in text.split(","): + raw = raw.strip() + if not raw: + continue + value = float(raw) + if value < 0.0 or value > 1.0: + raise ValueError("redaction ratios must be between 0 and 1") + values.append(value) + if not values: + raise ValueError("expected at least one ratio") + return values + + +def benchmark_matrix( + *, + entry_counts: Iterable[int], + payload_body_bytes: Iterable[int], + storage_modes: Iterable[str], + redaction_ratios: Iterable[float], + repeats: int, + warmups: int, +) -> dict[str, Any]: + if repeats < 1: + raise ValueError("repeats must be >= 1") + if warmups < 0: + raise ValueError("warmups must be >= 0") + entry_counts = list(entry_counts) + payload_body_bytes = list(payload_body_bytes) + storage_modes = list(storage_modes) + redaction_ratios = list(redaction_ratios) + + cases = [ + BenchmarkCase(count, size, mode, ratio) + for count in entry_counts + for size in payload_body_bytes + for mode in storage_modes + for ratio in redaction_ratios + ] + results = [_run_case(case, repeats=repeats, warmups=warmups) for case in cases] + return { + "schema": SCHEMA, + "generated_at": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()), + "python": sys.version.split()[0], + "platform": platform.platform(), + "parameters": { + "entry_counts": entry_counts, + "payload_body_bytes": payload_body_bytes, + "storage_modes": storage_modes, + "redaction_ratios": redaction_ratios, + "repeats": repeats, + "warmups": warmups, + }, + "cases": results, + } + + +def report_text(payload: dict[str, Any]) -> str: + lines = [ + "Forum deep-verify benchmark", + f"schema: {payload['schema']}", + f"python: {payload['python']} | platform: {payload['platform']}", + "", + ( + f"{'entries':>8} {'body':>8} {'storage':<12} {'redact':>7} " + f"{'build_ms':>10} {'chain_ms':>10} {'payload_ms':>11} {'deep_ms':>10}" + ), + ] + for case in payload["cases"]: + lines.append( + f"{case['entry_count']:>8} " + f"{case['payload_body_bytes']:>8} " + f"{case['storage_mode']:<12} " + f"{case['redaction_ratio']:>7.2f} " + f"{case['build']['ms']:>10.3f} " + f"{case['verify_chain']['mean_ms']:>10.3f} " + f"{case['verify_payloads']['mean_ms']:>11.3f} " + f"{case['verify_deep']['mean_ms']:>10.3f}" + ) + return "\n".join(lines) + + +def _run_case(case: BenchmarkCase, *, repeats: int, warmups: int) -> dict[str, Any]: + start = time.perf_counter_ns() + with _storage(case.storage_mode) as storage: + ledger = Ledger(storage, clock=_monotonic_clock()) + for seq in range(case.entry_count): + ledger.append( + actor="bench", + kind="payload", + payload=_payload(seq, case.payload_body_bytes), + causal_parent=None if seq == 0 else seq - 1, + ) + ledger.sync() + build_ms = _elapsed_ms(start) + entries = ledger.replay() + redacted = _redact_payloads(storage, entries, case.redaction_ratio) + + for _ in range(warmups): + ledger.verify() + ledger.verify_payloads() + ledger.verify(deep=True) + + chain = _time_repeated(ledger.verify, repeats) + payloads = _time_repeated(ledger.verify_payloads, repeats) + deep = _time_repeated(lambda: ledger.verify(deep=True), repeats) + + unique_payloads = len({entry.payload_hash for entry in entries}) + return { + "entry_count": case.entry_count, + "payload_body_bytes": case.payload_body_bytes, + "storage_mode": case.storage_mode, + "fsync_each": _fsync_each(case.storage_mode), + "redaction_ratio": case.redaction_ratio, + "unique_payloads": unique_payloads, + "payloads_present": unique_payloads - redacted, + "payloads_redacted": redacted, + "build": {"ms": build_ms}, + "verify_chain": chain, + "verify_payloads": payloads, + "verify_deep": deep, + } + + +def _payload(seq: int, payload_body_bytes: int) -> dict[str, str]: + prefix = f"{seq:016d}:" + fill = "x" * max(0, payload_body_bytes - len(prefix)) + return {"body": prefix + fill} + + +def _redact_payloads(storage: Storage, entries: list[LedgerEntry], ratio: float) -> int: + if ratio <= 0.0: + return 0 + hashes = sorted({entry.payload_hash for entry in entries}) + count = int(len(hashes) * ratio) + payloads = getattr(storage, "_payloads", None) + if not isinstance(payloads, dict): + return 0 + for payload_hash in hashes[:count]: + payloads.pop(payload_hash, None) + return count + + +def _time_repeated(fn, repeats: int) -> dict[str, Any]: + samples: list[float] = [] + result = None + for _ in range(repeats): + start = time.perf_counter_ns() + result = fn() + samples.append(_elapsed_ms(start)) + return { + "ok": bool(result), + "samples_ms": samples, + "mean_ms": statistics.fmean(samples), + "min_ms": min(samples), + "max_ms": max(samples), + } + + +def _elapsed_ms(start_ns: int) -> float: + return (time.perf_counter_ns() - start_ns) / 1_000_000.0 + + +def _monotonic_clock(): + tick = 0 + + def clock() -> float: + nonlocal tick + tick += 1 + return float(tick) + + return clock + + +def _fsync_each(storage_mode: str) -> bool | None: + if storage_mode == "memory": + return None + if storage_mode == "file-sync": + return True + if storage_mode == "file-batched": + return False + raise ValueError(f"unknown storage mode: {storage_mode}") + + +class _storage: + def __init__(self, mode: str) -> None: + if mode not in STORAGE_MODES: + raise ValueError(f"unknown storage mode: {mode}") + self._mode = mode + self._tmp: tempfile.TemporaryDirectory[str] | None = None + self.storage: Storage | None = None + + def __enter__(self) -> Storage: + if self._mode == "memory": + self.storage = InMemoryStorage() + return self.storage + self._tmp = tempfile.TemporaryDirectory(prefix="forum-deep-verify-") + fsync_each = _fsync_each(self._mode) + if fsync_each is None: + raise ValueError(f"file storage needs an fsync policy: {self._mode}") + self.storage = FileStorage(self._tmp.name, fsync_each=fsync_each) + return self.storage + + def __exit__(self, exc_type, exc, tb) -> None: + if self._tmp is not None: + self._tmp.cleanup() + + +def dumps(payload: dict[str, Any]) -> str: + return json.dumps(payload, indent=2) diff --git a/src/forum/cli.py b/src/forum/cli.py index 0777985..efec755 100644 --- a/src/forum/cli.py +++ b/src/forum/cli.py @@ -169,6 +169,78 @@ def _cmd_humanize(args) -> int: return 2 return 0 +def _cmd_import_trace(args) -> int: + import sys as _sys + + from forum.flight_recorder import TraceParseError, import_trace + + try: + text = _sys.stdin.read() if args.trace == "-" else open(args.trace, encoding="utf-8").read() + events = json.loads(text) + except (OSError, json.JSONDecodeError) as exc: + print(f"import-trace: cannot read trace: {exc}", file=_sys.stderr) + return 2 + try: + record = import_trace(events, args.format) + except TraceParseError as exc: + print(f"import-trace: {exc}", file=_sys.stderr) + return 2 + print(json.dumps(record, indent=2)) + return 0 + + +def _cmd_grade(args) -> int: + from forum.grade import grade_ledger + + led = _open_ledger(args.ledger) + print(json.dumps(grade_ledger(led, min_checks=args.min_checks), indent=2)) + return 0 + + +def _cmd_export_gradable(args) -> int: + from forum.gradable_export import gradable_record, write_gradable_jsonl + + led = _open_ledger(args.ledger) + row = gradable_record(led, min_checks=args.min_checks) + if args.out: + write_gradable_jsonl([row], args.out) + print(f"wrote 1 gradable-trajectory row -> {args.out} " + f"(grade={row['grade']['label']} reward={row['grade']['reward']})") + else: + print(json.dumps(row, indent=2)) + return 0 + + +def _cmd_mine(args) -> int: + """One command: fold ANY framework's trace into a verifiable ledger, grade it, + and append it as a gradable-trajectory datum. trace -> witnessed RL data.""" + import sys as _sys + + from forum.flight_recorder import TraceParseError, ledger_from_trace + from forum.gradable_export import gradable_record, write_gradable_jsonl + + try: + text = _sys.stdin.read() if args.trace == "-" else open(args.trace, encoding="utf-8").read() + events = json.loads(text) + except (OSError, json.JSONDecodeError) as exc: + print(f"mine: cannot read trace: {exc}", file=_sys.stderr) + return 2 + try: + led, _ = ledger_from_trace(events, args.format) + except TraceParseError as exc: + print(f"mine: {exc}", file=_sys.stderr) + return 2 + row = gradable_record(led, min_checks=args.min_checks) + if args.out: + write_gradable_jsonl([row], args.out) + print(f"mined {args.trace} -> {args.out} " + f"(grade={row['grade']['label']} reward={row['grade']['reward']}, " + f"merkle={row['oracle']['merkle_root'][:12]}...)") + else: + print(json.dumps(row, indent=2)) + return 0 + + def _cmd_route(args) -> int: from forum.roster import load_default from forum.route_frame import derive_route_frame, frame_payload @@ -625,6 +697,35 @@ def _cmd_bench(args) -> int: return 0 +def _cmd_bench_deep_verify(args) -> int: + from forum.bench_deep_verify import ( + benchmark_matrix, + dumps, + parse_float_csv, + parse_int_csv, + report_text, + ) + + try: + payload = benchmark_matrix( + entry_counts=parse_int_csv(args.entries), + payload_body_bytes=parse_int_csv(args.payload_bytes), + storage_modes=args.storage or ["memory"], + redaction_ratios=parse_float_csv(args.redaction_ratio), + repeats=args.repeats, + warmups=args.warmups, + ) + except ValueError as exc: + print(f"bench-deep-verify: {exc}", file=sys.stderr) + return 2 + text = dumps(payload) + if args.out: + with open(args.out, "w", encoding="utf-8") as fh: + fh.write(text + "\n") + print(text if args.json else report_text(payload)) + return 0 + + def _add_ledger(sp) -> None: sp.add_argument("--ledger", default=DEFAULT_LEDGER, help="ledger directory (default: forum-ledger)") @@ -699,6 +800,44 @@ def build_parser() -> argparse.ArgumentParser: ) route.set_defaults(func=_cmd_route) + ft = sub.add_parser( + "import-trace", + help="fold an external agent trace (LangSmith/OTel/AgentOps/generic JSON) " + "into a verifiable, replayable ledger — a flight recorder that " + "refutes its own tampering") + ft.add_argument("trace", help="path to a JSON trace (list of events), or - for stdin") + ft.add_argument("--format", choices=["langsmith", "otel", "agentops", "generic"], + default="generic") + ft.set_defaults(func=_cmd_import_trace) + + grade = sub.add_parser( + "grade", + help="grade a run's ledger — an outcome signal that CAN fail and counts " + "only independent checks (a producer cannot grade itself)") + grade.add_argument("ledger", help="path to a persisted ledger directory") + grade.add_argument("--min-checks", type=int, default=2) + grade.set_defaults(func=_cmd_grade) + + exp = sub.add_parser( + "export-gradable", + help="export a run's ledger as one forum.gradable-trajectory/1 datum " + "(prompt + trajectory + a can-fail grade + off-forum re-derivation inputs)") + exp.add_argument("ledger", help="path to a persisted ledger directory") + exp.add_argument("--out", help="append the sealed row to this JSONL (else print)") + exp.add_argument("--min-checks", type=int, default=2) + exp.set_defaults(func=_cmd_export_gradable) + + mine = sub.add_parser( + "mine", + help="one command: fold ANY framework's trace into a verifiable ledger, " + "grade it, and append it as witnessed gradable RL data") + mine.add_argument("trace", help="path to a JSON trace (list of events), or - for stdin") + mine.add_argument("--format", choices=["langsmith", "otel", "agentops", "generic"], + default="generic") + mine.add_argument("--out", help="append the sealed gradable row to this JSONL (else print)") + mine.add_argument("--min-checks", type=int, default=2) + mine.set_defaults(func=_cmd_mine) + submit = sub.add_parser("submit", help="plan and answer a request, witnessed") submit.add_argument("request") submit.add_argument("--max-model-calls", type=int, default=None, help="bound the run to N model calls (witnessed budget)") @@ -854,6 +993,30 @@ def build_parser() -> argparse.ArgumentParser: bench.add_argument("--json", action="store_true", help="emit both summaries and the delta as JSON") bench.set_defaults(func=_cmd_bench) + deep = sub.add_parser( + "bench-deep-verify", + help="measure ledger verify(deep=True) scaling across payload and storage variables", + ) + deep.add_argument("--entries", default="100,1000", help="comma-separated entry counts") + deep.add_argument("--payload-bytes", default="256,4096", help="comma-separated body byte targets") + deep.add_argument( + "--storage", + action="append", + choices=["memory", "file-sync", "file-batched"], + default=None, + help="storage mode to include; repeat for multiple modes", + ) + deep.add_argument( + "--redaction-ratio", + default="0,0.5,1", + help="comma-separated ratios of payload bodies removed before verification", + ) + deep.add_argument("--warmups", type=int, default=1, help="warmup iterations per case") + deep.add_argument("--repeats", type=int, default=5, help="measured iterations per case") + deep.add_argument("--json", action="store_true", help="emit the full JSON payload") + deep.add_argument("--out", default=None, help="write the full JSON payload to this path") + deep.set_defaults(func=_cmd_bench_deep_verify) + return parser diff --git a/src/forum/flight_recorder.py b/src/forum/flight_recorder.py new file mode 100644 index 0000000..c3afaa0 --- /dev/null +++ b/src/forum/flight_recorder.py @@ -0,0 +1,144 @@ +"""flight_recorder.py — fold any framework's agent trace into a verifiable ledger. + +Forum's differentiator is a hash-chained, replayable causal ledger. Every other +orchestration and observability tool (LangSmith, Langfuse, OTel, AgentOps) +records what an agent did, but the record is a mutable log you have to trust. +This is the seam that turns their trace into forum's: normalize an external +agent trace to ledger entries and append them, so ANY framework's run gains +tamper-evidence (verify), causal replay, and a Merkle root — a flight recorder +whose recording refutes its own tampering. + +Supported inputs (normalized to (actor, kind, payload, causal_parent)): + - "langsmith" LangChain/LangSmith runs: {name, run_type, inputs, outputs, + id, parent_run_id} + - "otel" OpenTelemetry spans: {name, span_id, parent_span_id, attributes} + - "agentops" AgentOps events: {event_type, ..., id, parent_id} + - "generic" already {actor, kind, payload, parent} (parent = index or None) + +FAIL LOUD: an event a mapper cannot read raises rather than silently dropping a +step — a flight recorder that quietly loses a step is worse than none. Causal +parents are resolved by id -> ledger seq; an unknown parent id is recorded as a +root (None) with the dangling id kept in the payload, never guessed. +""" +from __future__ import annotations + +from collections.abc import Sequence +from typing import Any + +from .ledger import GENESIS, InMemoryStorage, Ledger, merkle_root + + +class TraceParseError(ValueError): + """The input is not a trace we can fold faithfully into a ledger.""" + + +def _norm_langsmith(ev: dict) -> dict: + if "name" not in ev and "run_type" not in ev: + raise TraceParseError("langsmith event needs 'name' or 'run_type'") + return {"actor": str(ev.get("name") or ev.get("run_type") or "run"), + "kind": str(ev.get("run_type") or "run"), + "id": ev.get("id") or ev.get("run_id"), + "parent_id": ev.get("parent_run_id"), + "payload": ev} + + +def _norm_otel(ev: dict) -> dict: + if "name" not in ev and "span_id" not in ev: + raise TraceParseError("otel span needs 'name' or 'span_id'") + return {"actor": str(ev.get("name") or ev.get("span_id") or "span"), + "kind": "span", + "id": ev.get("span_id"), + "parent_id": ev.get("parent_span_id"), + "payload": ev} + + +def _norm_agentops(ev: dict) -> dict: + if "event_type" not in ev: + raise TraceParseError("agentops event needs 'event_type'") + return {"actor": str(ev.get("agent") or ev.get("event_type")), + "kind": str(ev["event_type"]), + "id": ev.get("id") or ev.get("event_id"), + "parent_id": ev.get("parent_id"), + "payload": ev} + + +def _norm_generic(ev: dict) -> dict: + if "actor" not in ev or "kind" not in ev: + raise TraceParseError("generic event needs 'actor' and 'kind'") + return {"actor": str(ev["actor"]), "kind": str(ev["kind"]), + "id": ev.get("id"), "parent_id": ev.get("parent"), + "payload": ev.get("payload", ev)} + + +_MAPPERS = {"langsmith": _norm_langsmith, "otel": _norm_otel, + "agentops": _norm_agentops, "generic": _norm_generic} + + +def normalize_trace(events: Sequence[dict], fmt: str) -> list[dict]: + mapper = _MAPPERS.get(fmt) + if mapper is None: + raise TraceParseError(f"unknown format {fmt!r}; known: {', '.join(sorted(_MAPPERS))}") + if not isinstance(events, list) or not events: + raise TraceParseError("trace must be a non-empty list of events") + return [mapper(ev if isinstance(ev, dict) else _bad(ev, i)) + for i, ev in enumerate(events)] + + +def _bad(ev: Any, i: int): + raise TraceParseError(f"event #{i} is not an object: {type(ev).__name__}") + + +def ledger_from_trace(events: Sequence[dict], fmt: str = "generic", + *, clock=None) -> tuple[Ledger, int]: + """Fold a trace into a fresh in-memory ledger and return (ledger, dangling). + This is the seam a downstream consumer (e.g. gradable export) needs: it wants + the ledger itself, not just a summary. Deterministic given `clock` (a counter + is used if none is supplied, so ts is reproducible). `dangling` counts events + whose parent id was unknown at append time (recorded as roots, id kept).""" + norm = normalize_trace(events, fmt) + counter = {"t": 0.0} + + def _c(): + counter["t"] += 1.0 + return counter["t"] + + ledger = Ledger(InMemoryStorage(), clock=clock or _c) + id_to_seq: dict[Any, int] = {} + dangling = 0 + for n in norm: + parent_seq = None + pid = n.get("parent_id") + if pid is not None: + parent_seq = id_to_seq.get(pid) + if parent_seq is None: + dangling += 1 # unknown parent -> root, id kept + e = ledger.append(actor=n["actor"], kind=n["kind"], + payload=n["payload"], causal_parent=parent_seq) + if n.get("id") is not None: + id_to_seq[n["id"]] = e.seq + return ledger, dangling + + +def import_trace(events: Sequence[dict], fmt: str = "generic", + *, clock=None) -> dict: + """Fold a trace into a fresh in-memory ledger and return a witnessed record: + the ledger entries, the verify verdict, and the Merkle root. Deterministic + given `clock` (a counter is used if none is supplied, so ts is reproducible).""" + ledger, dangling = ledger_from_trace(events, fmt, clock=clock) + replayed = ledger.replay() + entries = [{"seq": e.seq, "actor": e.actor, "kind": e.kind, + "causal_parent": e.causal_parent, "entry_hash": e.entry_hash} + for e in replayed] + hashes: list[str] = [e.entry_hash for e in replayed] + root = merkle_root(hashes) if hashes else GENESIS + return { + "schema": "forum.flight-recorder/1", + "format": fmt, + "entries": len(entries), + "verified": ledger.verify(deep=True), + "merkle_root": root, + "dangling_parents": dangling, + "ledger": entries, + "recheck": ("re-run import_trace over the same trace and compare the " + "merkle_root; tamper any entry and verify() fails"), + } diff --git a/src/forum/gradable_export.py b/src/forum/gradable_export.py new file mode 100644 index 0000000..2ac33fb --- /dev/null +++ b/src/forum/gradable_export.py @@ -0,0 +1,143 @@ +"""gradable_export.py — export a completed run as a witnessed, gradable datum. + +Forum's differentiator is a hash-chained, replayable ledger. This turns a +finished run (native, or one folded in from any framework by flight_recorder) +into one `forum.gradable-trajectory/1` row: a prompt, the trajectory, a grade +that CAN fail, and everything a consumer needs to re-derive both the integrity +and the grade OFF-forum with nothing but stdlib sha256 — no shared code, no +trust in forum. + +Honesty boundary (deliberate): this row does NOT assert a top-level +`witnessed:true`. That would be forum vouching for its own verification — the +theatre this whole line exists to avoid. The row carries only +`oracle.verified` (forum's OWN deep-verify at export, labeled as such) plus the +raw re-derivation inputs (`entries`, `merkle_root`, `grade_inputs`). The +WITNESSED verdict is computed by the consumer when it re-derives; a datum earns +"witnessed" by surviving that independent re-check, never by this exporter +stamping it. See trajectory_intake on the local-model side. + +Splits EXPORT out of flight_recorder.py (which only INGESTS a trace into a +ledger). Pure read over a Ledger; no clock, no network. +""" +from __future__ import annotations + +import hashlib +import json +from pathlib import Path +from typing import Any + +from .grade import grade_ledger +from .ledger import Ledger, merkle_root + + +def _task_id(request_text: str) -> str: + norm = " ".join((request_text or "").split()).lower() + return hashlib.sha256(norm.encode("utf-8")).hexdigest()[:16] + + +def _row_hash(row: dict[str, Any]) -> str: + """Same seal recipe the flywheel's task_curator uses: sha256 over the + sort_keys JSON of the row WITHOUT its own hash, first 16 hex. The two ends + agree on this recipe without importing each other.""" + body = {k: v for k, v in row.items() if k != "row_hash"} + return hashlib.sha256(json.dumps(body, sort_keys=True).encode()).hexdigest()[:16] + + +def _request_text(ledger: Ledger, override: str | None) -> str: + if override is not None: + return override + for e in ledger.query(kind="request"): + body = ledger.get_payload(e.payload_hash) + if isinstance(body.get("text"), str): + return body["text"] + return "" + + +def _final_answer(ledger: Ledger) -> str | None: + answer: str | None = None + for e in ledger.query(kind="result"): + body = ledger.get_payload(e.payload_hash) + # a synthesized final answer carries "answer" and is not a revision echo + if isinstance(body.get("answer"), str) and "revised_from" not in body: + answer = body["answer"] # keep the latest + return answer + + +def _model_calls(ledger: Ledger) -> int: + return sum(1 for e in ledger.query(kind="result") + if "model" in ledger.get_payload(e.payload_hash)) + + +def gradable_record(ledger: Ledger, *, request: str | None = None, + min_checks: int = 2) -> dict[str, Any]: + """Build one forum.gradable-trajectory/1 row from a finished run's ledger.""" + entries = ledger.replay() + projection = [{ + "seq": e.seq, + "ts": round(e.ts, 6), + "actor": e.actor, + "kind": e.kind, + "causal_parent": e.causal_parent, + "payload_hash": e.payload_hash, + "prev_hash": e.prev_hash, + "entry_hash": e.entry_hash, + } for e in entries] + hashes = [e.entry_hash for e in entries] + plans = ledger.query(kind="plan") + plan_hash = plans[-1].payload_hash if plans else None + grade = grade_ledger(ledger, min_checks=min_checks) + req_text = _request_text(ledger, request) + payload_bytes = sum(len(json.dumps(ledger.get_payload(e.payload_hash), + sort_keys=True)) for e in entries) + row: dict[str, Any] = { + "schema": "forum.gradable-trajectory/1", + "task_id": _task_id(req_text), + "prompt": req_text, + "trajectory": { + "plan_hash": plan_hash, + "answer": _final_answer(ledger), + "entries": projection, + }, + "grade": { + "reward": grade["reward"], + "label": grade["label"], + "checks": grade["checks"], + "refuted": grade["refuted"], + "producers": grade["producers"], + "graders": grade["graders"], + "derivation": grade["derivation"], + }, + "oracle": { + "kind": "ledger-rewitness", + "merkle_root": merkle_root(hashes), + # forum's OWN deep-verify at export — a self-report, NOT the witness. + # The consumer re-derives merkle_root + reward and decides MATCH/DRIFT. + "verified": ledger.verify(deep=True), + "grade_inputs": grade["grade_inputs"], + "recheck": ("recompute each entry_hash from its fields " + "(sha256 of seq|ts(.6f)|actor|kind|causal_parent|payload_hash|" + "prev_hash joined by \\x1f), fold the RFC6962 merkle " + "(0x00 leaf / 0x01 node, promote-odd) and compare to " + "merkle_root; recompute reward from grade_inputs. Tamper " + "any entry -> root mismatch; flip any grade_input.ok -> " + "reward changes."), + }, + "budget": { + "model_calls": _model_calls(ledger), + "payload_bytes": payload_bytes, + "entries": len(entries), + "clock": "injected", + }, + } + row["row_hash"] = _row_hash(row) + return row + + +def write_gradable_jsonl(records: list[dict[str, Any]], path: str | Path) -> int: + """Append records as sealed JSONL. Returns the number written.""" + p = Path(path) + p.parent.mkdir(parents=True, exist_ok=True) + with open(p, "a", encoding="utf-8") as f: + for row in records: + f.write(json.dumps(row, sort_keys=True) + "\n") + return len(records) diff --git a/src/forum/grade.py b/src/forum/grade.py new file mode 100644 index 0000000..8ef6ef7 --- /dev/null +++ b/src/forum/grade.py @@ -0,0 +1,84 @@ +"""grade.py — a deterministic OUTCOME grade for a completed run's ledger. + +Forum's ledger witnesses INTEGRITY: verify(deep=True) proves a trajectory is +untampered and replayable. It does NOT say whether the run SUCCEEDED. A training +loop that wants to admit a run as gradable RL data needs a success signal that +(a) can FAIL, and (b) is not authored by the same actor that produced the work. +This module derives exactly that, as a pure read over the ledger. + +The rule, and why each clause is load-bearing: + + - Only kinds {verdict, verification, intent_judgment} count as CHECKS. A + kind="result" is the producer's output, never its own grade. + - A check counts only if it is INDEPENDENT: its actor is not one of the actors + that produced a result in this run. A producer grading itself is discarded, + so a run cannot manufacture a passing grade for its own output. + - A check with ok is None (a verifier that failed to run) is non-informative: + it neither passes nor refutes, and is excluded from the count. It cannot be + laundered into either direction. + - reward = passed / (passed + refuted) over the independent, informative checks. + - label = PASS iff reward == 1.0 AND count >= min_checks AND refuted == 0; + UNVERIFIABLE iff count == 0 (integrity is NOT a success signal); + FAIL otherwise. + +UNVERIFIABLE is the honest floor: a run nobody independently checked is not +graded PASS just because its hash chain verifies. That is the whole point. +""" +from __future__ import annotations + +from typing import Any + +from .ledger import Ledger + +_CHECK_KINDS = ("verdict", "verification", "intent_judgment") + + +def _producer_actors(ledger: Ledger) -> set[str]: + """Actors that produced a result in this run. A grade authored by any of + these is self-graded and does not count toward an independent reward.""" + actors: set[str] = set() + for e in ledger.query(kind="result"): + actors.add(e.actor) + return actors + + +def grade_ledger(ledger: Ledger, *, min_checks: int = 2) -> dict[str, Any]: + """Grade a run purely from its ledger. Deterministic (no clock, no I/O).""" + producers = _producer_actors(ledger) + passed = 0 + refuted = 0 + inputs: list[dict[str, Any]] = [] + graders: list[str] = [] + for kind in _CHECK_KINDS: + for e in ledger.query(kind=kind): + if e.actor in producers: + continue # self-graded: not independent + ok = ledger.get_payload(e.payload_hash).get("ok") + if ok is None: + continue # verifier failed / non-informative: excluded + if ok: + passed += 1 + else: + refuted += 1 + inputs.append({"actor": e.actor, "ok": bool(ok), "kind": kind}) + graders.append(e.actor) + count = passed + refuted + reward = round(passed / count, 6) if count else 0.0 + if count == 0: + label = "UNVERIFIABLE" + elif reward == 1.0 and count >= min_checks and refuted == 0: + label = "PASS" + else: + label = "FAIL" + return { + "reward": reward, + "label": label, + "checks": count, + "refuted": refuted, + "producers": sorted(producers), + "graders": graders, + "grade_inputs": inputs, + "min_checks": min_checks, + "derivation": ("PASS iff reward==1.0 and checks>=min_checks and refuted==0; " + "UNVERIFIABLE iff checks==0; else FAIL"), + } diff --git a/tests/test_bench_deep_verify.py b/tests/test_bench_deep_verify.py new file mode 100644 index 0000000..2bc0582 --- /dev/null +++ b/tests/test_bench_deep_verify.py @@ -0,0 +1,45 @@ +import json + +from forum.bench_deep_verify import benchmark_matrix +from forum.cli import main + + +def test_deep_verify_benchmark_reports_payload_and_deep_timings(): + payload = benchmark_matrix( + entry_counts=[3], + payload_body_bytes=[32], + storage_modes=["memory"], + redaction_ratios=[0.0, 1.0], + repeats=1, + warmups=0, + ) + + assert payload["schema"] == "forum.deep-verify-benchmark/v1" + assert len(payload["cases"]) == 2 + assert payload["cases"][0]["verify_deep"]["ok"] is True + assert payload["cases"][0]["payloads_present"] == 3 + assert payload["cases"][1]["payloads_redacted"] == 3 + assert payload["cases"][1]["payloads_present"] == 0 + + +def test_bench_deep_verify_cli_json(capsys): + rc = main([ + "bench-deep-verify", + "--entries", + "2", + "--payload-bytes", + "16", + "--redaction-ratio", + "0", + "--warmups", + "0", + "--repeats", + "1", + "--json", + ]) + + payload = json.loads(capsys.readouterr().out) + assert rc == 0 + assert payload["schema"] == "forum.deep-verify-benchmark/v1" + assert payload["cases"][0]["entry_count"] == 2 + assert payload["cases"][0]["verify_payloads"]["ok"] is True diff --git a/tests/test_flight_recorder.py b/tests/test_flight_recorder.py new file mode 100644 index 0000000..7ffcee8 --- /dev/null +++ b/tests/test_flight_recorder.py @@ -0,0 +1,109 @@ +"""Falsifiers for the flight recorder — any framework's trace -> verifiable ledger. + +Load-bearing: (1) a trace folds into a ledger that VERIFIES (hash chain + +payloads); (2) tampering an entry breaks verification (the recording refutes its +own tampering); (3) causal parents are resolved by id -> seq; (4) each supported +format normalizes, and a malformed event fails LOUD (never silently dropped). +""" +from __future__ import annotations + +import pytest + +from forum.flight_recorder import ( + TraceParseError, + import_trace, + normalize_trace, +) +from forum.ledger import GENESIS, InMemoryStorage, Ledger + +GENERIC = [ + {"actor": "planner", "kind": "plan", "id": "a", "payload": {"goal": "x"}}, + {"actor": "worker", "kind": "act", "id": "b", "parent": "a", "payload": {"step": 1}}, + {"actor": "worker", "kind": "act", "id": "c", "parent": "b", "payload": {"step": 2}}, +] + + +def test_trace_folds_into_a_verifying_ledger(): + rec = import_trace(GENERIC, "generic") + assert rec["entries"] == 3 + assert rec["verified"] is True + assert rec["merkle_root"] != GENESIS + # deterministic: same trace -> same root + assert import_trace(GENERIC, "generic")["merkle_root"] == rec["merkle_root"] + + +def test_causal_parents_resolved_by_id(): + rec = import_trace(GENERIC, "generic") + b = next(e for e in rec["ledger"] if e["actor"] == "worker" and e["kind"] == "act" + and e["causal_parent"] == 0) + assert b["seq"] == 1 # b's parent is a (seq 0) + c = rec["ledger"][2] + assert c["causal_parent"] == 1 # c's parent is b (seq 1) + + +def test_tampering_breaks_verification(): + # rebuild the same ledger, tamper one stored entry, and verify() must fail + counter = {"t": 0.0} + led = Ledger(InMemoryStorage(), clock=lambda: (counter.__setitem__("t", counter["t"] + 1), counter["t"])[1]) + for ev in GENERIC: + led.append(actor=ev["actor"], kind=ev["kind"], payload=ev["payload"]) + assert led.verify(deep=True) + # corrupt a payload body in storage -> deep verify fails + entries = led._s.all() + bad_hash = "0" * 64 + object.__setattr__(entries[0], "payload_hash", bad_hash) + assert led.verify(deep=True) is False + + +def test_langsmith_and_otel_and_agentops_normalize(): + ls = normalize_trace([{"name": "chain", "run_type": "chain", "id": "1"}, + {"name": "llm", "run_type": "llm", "id": "2", "parent_run_id": "1"}], + "langsmith") + assert ls[0]["kind"] == "chain" and ls[1]["parent_id"] == "1" + ot = normalize_trace([{"name": "root", "span_id": "s1"}, + {"name": "child", "span_id": "s2", "parent_span_id": "s1"}], "otel") + assert ot[1]["parent_id"] == "s1" + ao = normalize_trace([{"event_type": "action", "id": "e1"}], "agentops") + assert ao[0]["kind"] == "action" + + +def test_langsmith_trace_imports_and_verifies(): + trace = [ + {"name": "agent", "run_type": "chain", "id": "r1", "inputs": {"q": "hi"}}, + {"name": "search", "run_type": "tool", "id": "r2", "parent_run_id": "r1"}, + {"name": "llm", "run_type": "llm", "id": "r3", "parent_run_id": "r1"}, + ] + rec = import_trace(trace, "langsmith") + assert rec["verified"] and rec["entries"] == 3 + assert rec["dangling_parents"] == 0 + + +def test_unknown_parent_is_a_root_not_a_guess(): + rec = import_trace([{"actor": "w", "kind": "act", "id": "b", "parent": "ghost", + "payload": {}}], "generic") + assert rec["ledger"][0]["causal_parent"] is None + assert rec["dangling_parents"] == 1 # counted, not hidden + + +def test_malformed_events_fail_loud(): + with pytest.raises(TraceParseError, match="non-empty list"): + import_trace([], "generic") + with pytest.raises(TraceParseError, match="unknown format"): + import_trace([{"actor": "a", "kind": "k"}], "nope") + with pytest.raises(TraceParseError): # generic needs actor+kind + import_trace([{"foo": "bar"}], "generic") + with pytest.raises(TraceParseError, match="not an object"): + import_trace([42], "generic") + + +def test_cli_import_trace(tmp_path, capsys): + import json + from types import SimpleNamespace + + from forum.cli import _cmd_import_trace + + f = tmp_path / "trace.json" + f.write_text(json.dumps(GENERIC), encoding="utf-8") + assert _cmd_import_trace(SimpleNamespace(trace=str(f), format="generic")) == 0 + rec = json.loads(capsys.readouterr().out) + assert rec["schema"] == "forum.flight-recorder/1" and rec["verified"] diff --git a/tests/test_gradable_export.py b/tests/test_gradable_export.py new file mode 100644 index 0000000..7263326 --- /dev/null +++ b/tests/test_gradable_export.py @@ -0,0 +1,106 @@ +"""Falsifiers for gradable_export.py — the exported datum must carry a grade +that can fail, must not launder integrity into success, must be deterministic, +and must expose everything needed to re-derive the witness off-forum. + +The pinned witness vector (test_pinned_witness_vector) fixes a known ledger's +merkle_root as a constant. The local-model consumer pins the SAME constant; if +either side's hash recipe drifts, one of the two vector tests fails loudly +(two independent implementations must agree, like emet's frozen vectors). +""" +from __future__ import annotations + +import json + +from forum.gradable_export import ( + _row_hash, + gradable_record, + write_gradable_jsonl, +) +from forum.ledger import GENESIS, InMemoryStorage, Ledger + +# The pinned cross-side vector. A ledger of exactly these three appends under the +# counter clock below has this merkle_root. The local-model intake test pins the +# identical value; a drift in either hash recipe breaks one of them. +PINNED_MERKLE_ROOT = "b79c96da02ece2a7ce60f75ff9611c7acfa30ea4d028015604e438bf7cfda1dc" + + +def _counter_ledger(): + counter = {"t": 0.0} + + def clock(): + counter["t"] += 1.0 + return counter["t"] + + return Ledger(InMemoryStorage(), clock=clock) + + +def _graded_run(): + led = _counter_ledger() + led.append(actor="client", kind="request", payload={"text": "sort a list"}) + led.append(actor="planner", kind="plan", payload={"tasks": ["T1"]}) + led.append(actor="worker", kind="result", payload={"id": "T1", "output": "def f(): ...", "ok": True, "model": "m"}) + led.append(actor="validator", kind="verdict", payload={"id": "T1", "ok": True, "score": 1.0}) + led.append(actor="verifier", kind="verification", payload={"ok": True, "detail": "", "source": "s"}) + led.append(actor="synthesizer", kind="result", payload={"answer": "sorted"}) + return led + + +def test_graded_run_exports_pass_record(): + row = gradable_record(_graded_run()) + assert row["schema"] == "forum.gradable-trajectory/1" + assert row["prompt"] == "sort a list" + assert row["trajectory"]["answer"] == "sorted" + assert row["grade"]["label"] == "PASS" + assert row["grade"]["reward"] == 1.0 + assert row["oracle"]["merkle_root"] != GENESIS + assert row["oracle"]["verified"] is True + assert row["_"] if False else _row_hash(row) == row["row_hash"] # seal is self-consistent + + +def test_unchecked_run_is_unverifiable(): + # a run with a producer and a final answer but ZERO independent checks: + # integrity verifies, but the grade must be UNVERIFIABLE, never PASS. + led = _counter_ledger() + led.append(actor="client", kind="request", payload={"text": "do a thing"}) + led.append(actor="worker", kind="result", payload={"id": "T1", "output": "x", "ok": True}) + led.append(actor="synthesizer", kind="result", payload={"answer": "done"}) + row = gradable_record(led) + assert row["oracle"]["verified"] is True # integrity holds + assert row["grade"]["label"] == "UNVERIFIABLE" # but it is NOT graded PASS + assert "witnessed" not in row # forum never stamps the witness + + +def test_export_is_deterministic(): + a = gradable_record(_graded_run()) + b = gradable_record(_graded_run()) + assert json.dumps(a, sort_keys=True) == json.dumps(b, sort_keys=True) + assert a["row_hash"] == b["row_hash"] + assert a["oracle"]["merkle_root"] == b["oracle"]["merkle_root"] + + +def test_grade_inputs_expose_reward_derivation(): + row = gradable_record(_graded_run()) + inputs = row["oracle"]["grade_inputs"] + assert all("actor" in i and "ok" in i and "kind" in i for i in inputs) + passed = sum(1 for i in inputs if i["ok"]) + total = len(inputs) + assert round(passed / total, 6) == row["grade"]["reward"] + + +def test_write_jsonl_roundtrips(tmp_path): + row = gradable_record(_graded_run()) + out = tmp_path / "data.jsonl" + n = write_gradable_jsonl([row], out) + assert n == 1 + loaded = [json.loads(line) for line in out.read_text().splitlines() if line.strip()] + assert loaded[0]["row_hash"] == row["row_hash"] + + +def test_pinned_witness_vector(): + # freeze the known ledger's merkle_root so the cross-side recipe can't drift + root = gradable_record(_graded_run())["oracle"]["merkle_root"] + if PINNED_MERKLE_ROOT != "PLACEHOLDER": + assert root == PINNED_MERKLE_ROOT, ( + f"merkle recipe drifted: {root} != pinned {PINNED_MERKLE_ROOT}") + # else: first run — capture the value below and pin it on both sides + assert len(root) == 64 and root != GENESIS diff --git a/tests/test_grade.py b/tests/test_grade.py new file mode 100644 index 0000000..575776c --- /dev/null +++ b/tests/test_grade.py @@ -0,0 +1,95 @@ +"""Falsifiers for grade.py — the outcome grade must be able to FAIL, and a +producer must never be able to author its own passing grade. + +Load-bearing: (1) flipping a passing check to failing flips the grade (can-fail); +(2) a check authored by the producing actor is not independent -> UNVERIFIABLE, +never PASS; (3) a run with zero checks is UNVERIFIABLE, so integrity is not +laundered into success; (4) the grade is deterministic under an injected clock. +""" +from __future__ import annotations + +from forum.grade import grade_ledger +from forum.ledger import InMemoryStorage, Ledger + + +def _ledger(): + # deterministic clock so ts is reproducible + counter = {"t": 0.0} + + def clock(): + counter["t"] += 1.0 + return counter["t"] + + return Ledger(InMemoryStorage(), clock=clock) + + +def test_flipped_verdict_flips_grade(): + # a producer result plus two independent passing verdicts -> PASS reward 1.0 + led = _ledger() + led.append(actor="worker", kind="result", payload={"id": "T1", "output": "x", "ok": True}) + led.append(actor="validator", kind="verdict", payload={"id": "T1", "ok": True, "score": 1.0}) + led.append(actor="verifier", kind="verification", payload={"ok": True, "detail": "", "source": "s"}) + g = grade_ledger(led, min_checks=2) + assert g["label"] == "PASS", g + assert g["reward"] == 1.0 + + # rebuild flipping the verdict ok -> reward < 1.0, label FAIL (the grade CAN fail) + led2 = _ledger() + led2.append(actor="worker", kind="result", payload={"id": "T1", "output": "x", "ok": True}) + led2.append(actor="validator", kind="verdict", payload={"id": "T1", "ok": False, "score": 0.0}) + led2.append(actor="verifier", kind="verification", payload={"ok": True, "detail": "", "source": "s"}) + g2 = grade_ledger(led2, min_checks=2) + assert g2["label"] == "FAIL", g2 + assert g2["reward"] < 1.0 + assert g2["refuted"] == 1 + + +def test_self_check_is_unverifiable(): + # the only check is authored by the producing actor -> not independent -> UNVERIFIABLE + led = _ledger() + led.append(actor="worker", kind="result", payload={"id": "T1", "output": "x", "ok": True}) + led.append(actor="worker", kind="verdict", payload={"id": "T1", "ok": True, "score": 1.0}) + g = grade_ledger(led) + assert g["label"] == "UNVERIFIABLE", g + assert g["checks"] == 0 + + +def test_zero_checks_is_unverifiable_not_pass(): + # integrity without a grade must never read as success + led = _ledger() + led.append(actor="worker", kind="result", payload={"id": "T1", "output": "x", "ok": True}) + led.append(actor="synthesizer", kind="result", payload={"answer": "done"}) + g = grade_ledger(led) + assert g["label"] == "UNVERIFIABLE" + assert g["reward"] == 0.0 + + +def test_none_ok_verification_is_non_informative(): + # a verifier that failed to run (ok=None) is excluded, not counted as pass or fail + led = _ledger() + led.append(actor="worker", kind="result", payload={"id": "T1", "output": "x", "ok": True}) + led.append(actor="verifier", kind="verification", payload={"ok": None, "detail": "verifier crashed", "source": ""}) + led.append(actor="validator", kind="verdict", payload={"id": "T1", "ok": True, "score": 1.0}) + g = grade_ledger(led, min_checks=1) + assert g["checks"] == 1 # only the verdict counts + assert g["label"] == "PASS" + + +def test_below_min_checks_is_not_pass(): + led = _ledger() + led.append(actor="worker", kind="result", payload={"id": "T1", "output": "x", "ok": True}) + led.append(actor="validator", kind="verdict", payload={"id": "T1", "ok": True, "score": 1.0}) + g = grade_ledger(led, min_checks=2) + assert g["checks"] == 1 + assert g["label"] == "FAIL" # one passing check, but min_checks=2 not met + + +def test_grade_is_deterministic(): + def build(): + led = _ledger() + led.append(actor="worker", kind="result", payload={"id": "T1", "output": "x", "ok": True}) + led.append(actor="validator", kind="verdict", payload={"id": "T1", "ok": True, "score": 1.0}) + led.append(actor="verifier", kind="verification", payload={"ok": True, "detail": "", "source": "s"}) + return grade_ledger(led) + + assert build() == build() diff --git a/tests/test_mine_cli.py b/tests/test_mine_cli.py new file mode 100644 index 0000000..b7c8ac6 --- /dev/null +++ b/tests/test_mine_cli.py @@ -0,0 +1,45 @@ +"""Falsifier for the one-command `forum mine` path: any framework's trace folds +into a verifiable ledger, gets graded, and is appended as a sealed +gradable-trajectory datum — in a single command, reusing the flight recorder. +""" +from __future__ import annotations + +import json + +from forum.cli import main + + +def test_mine_generic_trace_to_gradable_jsonl(tmp_path, capsys): + # a generic trace: a producer result plus two independent checks -> gradable PASS + trace = [ + {"actor": "client", "kind": "request", "id": "r", "payload": {"text": "add two ints"}}, + {"actor": "worker", "kind": "result", "id": "t1", "parent": "r", + "payload": {"id": "t1", "output": "def add(a,b): return a+b", "ok": True, "model": "m"}}, + {"actor": "validator", "kind": "verdict", "id": "v", "parent": "t1", + "payload": {"id": "t1", "ok": True, "score": 1.0}}, + {"actor": "verifier", "kind": "verification", "id": "w", "parent": "t1", + "payload": {"ok": True, "detail": "runs", "source": "s"}}, + ] + trace_path = tmp_path / "trace.json" + trace_path.write_text(json.dumps(trace), encoding="utf-8") + out = tmp_path / "data.jsonl" + + rc = main(["mine", str(trace_path), "--format", "generic", "--out", str(out)]) + assert rc == 0 + + rows = [json.loads(line) for line in out.read_text().splitlines() if line.strip()] + assert len(rows) == 1 + row = rows[0] + assert row["schema"] == "forum.gradable-trajectory/1" + assert row["prompt"] == "add two ints" + assert row["grade"]["label"] == "PASS" + assert row["grade"]["reward"] == 1.0 + assert row["oracle"]["merkle_root"] and len(row["oracle"]["merkle_root"]) == 64 + # forum never stamps a top-level witnessed claim; the grade inputs are exposed + assert "witnessed" not in row + assert len(row["oracle"]["grade_inputs"]) == 2 + + # appending a second mine grows the dataset (append semantics) + main(["mine", str(trace_path), "--format", "generic", "--out", str(out)]) + rows2 = [line for line in out.read_text().splitlines() if line.strip()] + assert len(rows2) == 2