diff --git a/docs/EVIDENCE.md b/docs/EVIDENCE.md index 2fe5fca..c410ba9 100644 --- a/docs/EVIDENCE.md +++ b/docs/EVIDENCE.md @@ -151,15 +151,17 @@ where the claim is derived: ▼ ┌─────────────┼─────────────┐ ▼ ▼ ▼ - ┌───────┐ ┌──────────┐ ┌─────────┐ - │ diff │ │ inspect │ │ archive │ - └───────┘ └──────────┘ └─────────┘ + ┌────────┐ ┌──────────┐ ┌──────┐ + │ replay │ │ inspect │ │ diff │ + └────────┘ └──────────┘ └──────┘ ``` Generation flows forward only. Preservation is read-only after save. -The arrow from artifact into `diff`/`inspect`/`archive` represents +The arrows from the artifact into `replay`/`inspect`/`diff` represent consumers reading the preserved evidence — they do not re-derive the -claim, they re-present it. +claim, they re-present it. (These three are the read operations +detailed in §9; the full consumer surface also includes `history` / +`timeline` / `matrix` / `verify` / `export`.) --- diff --git a/falsifyai/cli/render.py b/falsifyai/cli/render.py index e519a68..6317fe1 100644 --- a/falsifyai/cli/render.py +++ b/falsifyai/cli/render.py @@ -1,19 +1,21 @@ -"""Plain-text terminal output for ``falsifyai run``, ``replay``, and ``diff``. +"""Plain-text terminal output for the human-readable CLI surfaces. -MVP scope: one row per case + a summary footer + the session id and store -path so the user can find their saved artifact. No colors, no boxes, no -JSON. Rich/colored output and ``--json`` land in Week 3 per -[plan.md section 22.1](../../plan.md). +Deliberately plain text -- no colors, no boxes, no JSON. The discipline is +evidence density, not presentation: one row per case + a summary footer + +the session id and store path so the user can find their saved artifact. +This module renders ``run`` / ``replay`` (``render_session``), ``diff`` +(``render_diff``), ``verify`` (``render_verify`` / ``render_verify_all``), +and ``export`` (``render_export`` / ``render_export_refusal``). The ``loaded_from`` parameter on ``render_session`` is what distinguishes the replay path: when set, an extra header line indicates the user is looking at a stored session rather than a fresh run. The detection of -legacy artifacts (pre-PR-11, no CI evidence) lives in this module too -- +legacy artifacts (no CI evidence preserved) lives in this module too -- the artifact shape, not the consumer, determines what's renderable. -``render_diff`` is the diff CLI's render path (PR #14). It consumes a -``DiffReport`` (consumer-side dataclass from cli/diff.py) and prints a -compressed transition table: only rows where something changed are shown. +``render_diff`` consumes a ``DiffReport`` (consumer-side dataclass from +cli/diff.py) and prints a compressed transition table: only rows where +something changed are shown. """ import sys diff --git a/falsifyai/replay/protocol.py b/falsifyai/replay/protocol.py index 7840341..ca4ce25 100644 --- a/falsifyai/replay/protocol.py +++ b/falsifyai/replay/protocol.py @@ -1,12 +1,15 @@ """ReplayStore Protocol and its exception hierarchy. The Protocol is the contract every store implementation must satisfy. Two -impls ship in PR #6: ``InMemoryStore`` (ephemeral, also useful as a test -double) and ``SQLiteStore`` (default; file-backed at ``.falsifyai/replays.db``). - -MVP surface only — ``case_history`` and ``diff_sessions`` from plan.md -section 18.1 are deferred to the Week 2 PRs that introduce ``falsifyai diff`` -and verdict-history queries. +impls ship: ``InMemoryStore`` (ephemeral, also useful as a test double) and +``SQLiteStore`` (default; file-backed at ``.falsifyai/replays.db``). + +The surface is deliberately small — ``save_session`` / ``load_session`` / +``query_sessions`` / ``close``. Verdict-history and cross-session views +(``history`` / ``timeline`` / ``matrix``) and ``diff`` are consumer-side +operations layered on ``query_sessions`` and ``load_session``; they are not +extra store methods. Keeping the store interface minimal is what lets new +consumer commands ship without widening the persistence contract. """ from collections.abc import Iterator diff --git a/tests/meta/test_evidence_doc_freshness.py b/tests/meta/test_evidence_doc_freshness.py index 6f3e398..cc4b726 100644 --- a/tests/meta/test_evidence_doc_freshness.py +++ b/tests/meta/test_evidence_doc_freshness.py @@ -5,7 +5,7 @@ it. Stale claims there are higher-risk than stale claims anywhere else in the repo, because the document is the thing external readers trust. -This is a **coarse** freshness guard, not a prose validator. It enforces two +This is a **coarse** freshness guard, not a prose validator. It enforces three cheap, high-signal invariants and nothing semantic: 1. Every ``Verdict`` enum member is named at least once in the document. This @@ -18,10 +18,17 @@ 2. The document names the read-only preservation consumers at least once, so the producer/consumer boundary the artifact depends on stays documented as the command surface grows. +3. Every leaf consumer box in the §3 lifecycle diagram names a real CLI + command. This catches the specific drift where the diagram keeps a node for + an operation that was never (or no longer) a command -- e.g. a stale + ``archive`` box surviving long after the consumer surface settled on + ``replay`` / ``inspect`` / ``diff``. The prose word "archive" is allowed to + appear elsewhere (retention discussion); only diagram *boxes* are checked. If this test fails, update ``docs/EVIDENCE.md`` -- do not weaken the assertion. """ +import re from pathlib import Path from falsifyai.verdict.models import Verdict @@ -43,11 +50,54 @@ ) +# Every CLI command FalsifyAI ships. A leaf box in the lifecycle diagram must +# name one of these; "archive" (a past stale node) is intentionally absent. +_ALL_COMMANDS = frozenset( + { + "run", + "replay", + "inspect", + "diff", + "history", + "timeline", + "matrix", + "minimize", + "export", + "verify", + "doctor", + } +) + +# A box-drawing cell holding a single lowercase token: ``│ replay │``. The +# inner whitespace is horizontal-only (``[^\S\n]`` not ``\s``) so a token never +# pairs an opening ``│`` on one line with a closing ``│`` on the next -- that +# cross-line greediness would otherwise capture a trailing label like the +# ``│ evidence`` annotation beside the artifact box. Matched non-overlapping +# so a one-line ``│ replay │ │ inspect │ │ diff │`` row yields all three. +# Multi-word boxes ("the durable record") never match: ``[a-z_]+`` stops at the +# first space and the following ``│`` is not adjacent. +_DIAGRAM_BOX = re.compile(r"│[^\S\n]*([a-z_]+)[^\S\n]*│") + + def _doc_text() -> str: assert _EVIDENCE_DOC.exists(), f"evidence contract doc missing: {_EVIDENCE_DOC}" return _EVIDENCE_DOC.read_text(encoding="utf-8") +def _lifecycle_diagram() -> str: + """The fenced ASCII block in §3 -- the one containing the artifact node.""" + blocks = _doc_text().split("```") + # Fenced content sits at odd indices once split on the fence marker. + for block in blocks[1::2]: + if "REPLAY ARTIFACT" in block: + return block + raise AssertionError( + "lifecycle diagram not found in docs/EVIDENCE.md: no fenced block names " + "'REPLAY ARTIFACT'. The §3 diagram is the artifact contract's visual " + "anchor -- if it was removed or renamed, update this guard deliberately." + ) + + def test_every_verdict_appears_in_evidence_doc() -> None: text = _doc_text() missing = [v.name for v in Verdict if v.name not in text] @@ -67,3 +117,24 @@ def test_evidence_doc_names_readonly_consumers() -> None: f"The producer/consumer boundary is load-bearing protocol semantics; " f"keep the consumer surface documented." ) + + +def test_lifecycle_diagram_leaf_consumers_are_real_commands() -> None: + # Leaf consumer boxes fan out *below* the artifact node; restrict the scan + # to that region so upstream pipeline boxes ("spec", etc.) are not checked. + _, _, leaf_region = _lifecycle_diagram().partition("REPLAY ARTIFACT") + boxes = _DIAGRAM_BOX.findall(leaf_region) + + assert boxes, ( + "no leaf consumer boxes found below the REPLAY ARTIFACT node in the " + "docs/EVIDENCE.md lifecycle diagram. The diagram must show the consumers " + "reading the preserved artifact; if its shape changed, update this guard." + ) + + unknown = sorted({b for b in boxes if b not in _ALL_COMMANDS}) + assert not unknown, ( + f"lifecycle diagram in docs/EVIDENCE.md names non-command leaf nodes: " + f"{unknown}. Every consumer box below the artifact node must be a real " + f"CLI command -- this is the guard that locks out the stale 'archive' " + f"node. Known commands: {sorted(_ALL_COMMANDS)}." + ) diff --git a/tests/meta/test_readme_command_surface.py b/tests/meta/test_readme_command_surface.py new file mode 100644 index 0000000..b1d69b3 --- /dev/null +++ b/tests/meta/test_readme_command_surface.py @@ -0,0 +1,52 @@ +"""Stale-doc tripwire: every CLI subcommand is referenced in the README. + +The README is the public face of the CLI; its command reference is what a new +user reads to learn the surface. The failure mode this locks out is silent: +a new subcommand is wired into ``build_parser`` and ships, but the README is +never updated to mention it, so the documented surface drifts behind the real +one. + +This is a **coarse** guard, deliberately. It asserts only that the literal +token ``falsifyai `` appears somewhere in ``README.md`` for every +registered subcommand. It does not validate that the surrounding prose is +correct, complete, or current -- that is review's job. Catching the +"command exists but is undocumented" drift cheaply is the whole ambition. + +The subcommand list is read from ``build_parser`` itself, not hardcoded, so +the guard tracks the real surface automatically as commands are added. + +If this test fails, add the command to ``README.md`` -- do not weaken the +assertion. +""" + +import argparse +from pathlib import Path + +from falsifyai.cli.main import build_parser + +_README = Path(__file__).resolve().parents[2] / "README.md" + + +def _subcommand_names() -> list[str]: + parser = build_parser() + for action in parser._actions: + if isinstance(action, argparse._SubParsersAction): + return sorted(action.choices) + raise AssertionError( + "build_parser() exposes no subparsers; the CLI dispatch shape changed. " + "Update this guard to match how subcommands are now registered." + ) + + +def test_every_subcommand_is_referenced_in_readme() -> None: + readme = _README.read_text(encoding="utf-8") + commands = _subcommand_names() + assert commands, "no subcommands discovered from build_parser()" + + missing = [c for c in commands if f"falsifyai {c}" not in readme] + assert not missing, ( + f"README.md does not reference these CLI subcommands: {missing}. " + f"A command is registered in build_parser() but the README command " + f"reference never mentions ``falsifyai `` for it. Document the " + f"command before shipping." + ) diff --git a/tests/meta/test_release_gate_coverage_sync.py b/tests/meta/test_release_gate_coverage_sync.py new file mode 100644 index 0000000..e2bce57 --- /dev/null +++ b/tests/meta/test_release_gate_coverage_sync.py @@ -0,0 +1,62 @@ +"""Stale-config tripwire: the release gate's coverage floor matches CI's. + +``scripts/release_gate.py`` runs ``pytest --cov-fail-under=COVERAGE_FLOOR`` and +``.github/workflows/ci.yml`` runs ``pytest ... --cov-fail-under=N``. These are +two independently hardcoded copies of the same number, kept in step today only +by a hand-maintained "keep in sync" comment in the CI workflow. That is exactly +the kind of duplicated constant that drifts silently: bump one, forget the +other, and the release tooling no longer enforces what CI enforces (or vice +versa). + +This guard makes the drift fail loudly. It parses both numbers and asserts +equality -- nothing more. It does not opine on what the floor *should* be; it +only refuses to let the two copies disagree. + +If this test fails, reconcile ``COVERAGE_FLOOR`` in scripts/release_gate.py with +``--cov-fail-under`` in .github/workflows/ci.yml so they are equal again. +""" + +import re +from pathlib import Path + +_ROOT = Path(__file__).resolve().parents[2] +_RELEASE_GATE = _ROOT / "scripts" / "release_gate.py" +_CI_WORKFLOW = _ROOT / ".github" / "workflows" / "ci.yml" + + +def _release_gate_floor() -> int: + text = _RELEASE_GATE.read_text(encoding="utf-8") + match = re.search(r"^COVERAGE_FLOOR\s*=\s*(\d+)", text, re.MULTILINE) + assert match, ( + "could not find a top-level ``COVERAGE_FLOOR = `` in " + "scripts/release_gate.py; the release-gate coverage constant moved or " + "was renamed. Update this guard to read it from its new home." + ) + return int(match.group(1)) + + +def _ci_floor() -> int: + text = _CI_WORKFLOW.read_text(encoding="utf-8") + matches = re.findall(r"--cov-fail-under=(\d+)", text) + assert matches, ( + "could not find ``--cov-fail-under=`` in .github/workflows/ci.yml; " + "the CI coverage gate moved or was renamed. Update this guard to read it " + "from its new home." + ) + distinct = set(matches) + assert len(distinct) == 1, ( + f".github/workflows/ci.yml uses multiple --cov-fail-under values " + f"{sorted(distinct)}; the CI coverage floor is no longer a single number, " + f"so 'matches release gate' is ambiguous. Reconcile CI first." + ) + return int(matches[0]) + + +def test_release_gate_coverage_floor_matches_ci() -> None: + gate = _release_gate_floor() + ci = _ci_floor() + assert gate == ci, ( + f"coverage floor drift: scripts/release_gate.py COVERAGE_FLOOR={gate} but " + f".github/workflows/ci.yml --cov-fail-under={ci}. Release tooling and CI " + f"must enforce the same coverage floor. Reconcile the two constants." + ) diff --git a/tests/unit/test_bundle_writer.py b/tests/unit/test_bundle_writer.py index 962e3f7..1585723 100644 --- a/tests/unit/test_bundle_writer.py +++ b/tests/unit/test_bundle_writer.py @@ -1,8 +1,7 @@ """Unit tests for ``falsifyai.bundle.writer`` (PR-32, Phase B). -RED phase: imports ``BundleManifest``, ``BundleFileEntry``, ``write_bundle``, -and the schema constants which do not exist yet. After GREEN, all tests -should pass. +Exercises ``BundleManifest``, ``BundleFileEntry``, ``write_bundle``, and the +schema constants. Tests grouped by acceptance criterion from PR-32 plan §5: diff --git a/tests/unit/test_cli_diff_sharpening.py b/tests/unit/test_cli_diff_sharpening.py index 736fd46..2e5a9d1 100644 --- a/tests/unit/test_cli_diff_sharpening.py +++ b/tests/unit/test_cli_diff_sharpening.py @@ -1,12 +1,8 @@ -"""RED-phase tests for PR-28 diff-sharpening: --strict, --show-timeline, exit 6. +"""Tests for diff-sharpening: --strict, --show-timeline, exit 6. All tests in this file import ``STRICT_CONFIDENCE_DROP_THRESHOLD`` and -``LOW_FALSIFIABILITY_THRESHOLD`` from ``falsifyai.cli.diff``. Those constants -do not exist yet, so the entire module fails to collect in RED phase. Once -GREEN adds the constants and the flag logic, every test here should pass. - -Tests that already document a true property (introspection, exit-code parity -with no flags) will pass immediately after import is unblocked. +``LOW_FALSIFIABILITY_THRESHOLD`` from ``falsifyai.cli.diff`` and exercise the +flag logic those constants drive, alongside the exit-code parity with no flags. Covers acceptance criteria §5 of dev_notes/plans/PR-28-diff-sharpening.md: - Named constants present with correct values @@ -24,8 +20,8 @@ from falsifyai.cli import diff as diff_module from falsifyai.cli.diff import ( - LOW_FALSIFIABILITY_THRESHOLD, # RED: ImportError until GREEN adds this - STRICT_CONFIDENCE_DROP_THRESHOLD, # RED: ImportError until GREEN adds this + LOW_FALSIFIABILITY_THRESHOLD, + STRICT_CONFIDENCE_DROP_THRESHOLD, ) from falsifyai.replay.in_memory_store import InMemoryStore from falsifyai.replay.models import CaseResult, ReplayArtifact, SessionVerdict diff --git a/tests/unit/test_cli_history.py b/tests/unit/test_cli_history.py index 667c069..9bc5798 100644 --- a/tests/unit/test_cli_history.py +++ b/tests/unit/test_cli_history.py @@ -4,7 +4,7 @@ Decisions A1/B1/D1/E1/F1/G1 + X1/Y1/Z1 from ``dev_notes/plans/PR-24-falsifyai-history-cli.md``. -RED phase: these tests describe the public surface before it exists. +These tests exercise the public surface of ``falsifyai.cli.history``. """ import argparse diff --git a/tests/unit/test_cli_inspect.py b/tests/unit/test_cli_inspect.py index 02004fa..903cf70 100644 --- a/tests/unit/test_cli_inspect.py +++ b/tests/unit/test_cli_inspect.py @@ -5,8 +5,8 @@ verdict. Tests below enforce that rule and the decisions A-F from `dev_notes/plans/PR-19-falsifyai-inspect-cli.md`. -RED phase: these tests describe the public surface before it exists. They -are expected to fail until cli/inspect.py is implemented. +These tests exercise the public surface of ``cli/inspect.py`` and enforce the +never-re-derive-a-verdict rule. """ import argparse diff --git a/tests/unit/test_cli_invocation_model.py b/tests/unit/test_cli_invocation_model.py index 9a18e36..b6653f2 100644 --- a/tests/unit/test_cli_invocation_model.py +++ b/tests/unit/test_cli_invocation_model.py @@ -1,7 +1,7 @@ """Unit tests for ``CliInvocation`` and the ``ReplayArtifact.cli_invocation`` field (PR-35). -RED phase: imports ``CliInvocation`` from ``falsifyai.replay.models`` which -does not exist yet. After GREEN, all tests should pass. +Exercises ``CliInvocation`` from ``falsifyai.replay.models`` and the +``ReplayArtifact.cli_invocation`` field. Capture contract (per PR-35 plan §1): - ``CliInvocation`` is a frozen dataclass with exactly two fields: diff --git a/tests/unit/test_cli_run_captures_invocation.py b/tests/unit/test_cli_run_captures_invocation.py index 3c2dc82..301e75f 100644 --- a/tests/unit/test_cli_run_captures_invocation.py +++ b/tests/unit/test_cli_run_captures_invocation.py @@ -1,7 +1,7 @@ """Unit tests for ``cmd_run``'s ``cli_invocation`` capture (PR-35, Phase C). -RED phase: the capture helper and the field-attachment do not exist yet. -After GREEN, all tests should pass. +Exercises the ``cli_invocation`` capture helper and its field-attachment in +``cmd_run``. Capture contract (per PR-35 plan §1): - Capture at entry into ``cmd_run`` (single capture point) diff --git a/tests/unit/test_cli_store_lifecycle.py b/tests/unit/test_cli_store_lifecycle.py index 7c1eb59..2b00e53 100644 --- a/tests/unit/test_cli_store_lifecycle.py +++ b/tests/unit/test_cli_store_lifecycle.py @@ -7,11 +7,14 @@ later unlink of the database file — is the failure this locks out. This is a single parametrized harness rather than a close assertion duplicated -across eight command test files. ``run`` (and ``minimize``) are intentionally -absent: they are *producers* that orchestrate generation -> interpretation -> -preservation, not read-only consumers, and their store lifecycle is asserted in -their own test file where the execution stack is already mocked. Folding them in -here would blur the producer/consumer boundary the harness exists to protect. +across eight command test files. ``run`` is intentionally absent: it is a +*producer* that orchestrates generation -> interpretation -> preservation, not a +read-only consumer, and its store lifecycle is asserted in its own test file +where the execution stack is already mocked. ``minimize`` is also absent for a +different reason -- it never opens a ``ReplayStore`` at all (it runs an +in-memory search and persists nothing), so there is no store lifecycle to pin. +Folding either in here would blur the producer/consumer boundary the harness +exists to protect. Two situations matter, for every consumer: diff --git a/tests/unit/test_integrity_checks.py b/tests/unit/test_integrity_checks.py index 3e4afbf..45d598d 100644 --- a/tests/unit/test_integrity_checks.py +++ b/tests/unit/test_integrity_checks.py @@ -1,7 +1,6 @@ """Unit tests for ``falsifyai.integrity.checks`` (PR-31). -RED phase: imports CheckResult, CheckStatus, IntegrityReport, run_integrity_checks -which do not exist yet. After GREEN, all tests should pass. +Exercises CheckResult, CheckStatus, IntegrityReport, and run_integrity_checks. Eight integrity checks per PR-31 plan §3: diff --git a/tests/unit/test_paraphrase.py b/tests/unit/test_paraphrase.py index 4c02d27..6a07627 100644 --- a/tests/unit/test_paraphrase.py +++ b/tests/unit/test_paraphrase.py @@ -5,7 +5,7 @@ ``max_attempts`` per requested paraphrase. Phase B of the validation campaign — see ``dev_notes/plans/PR-22-paraphrase-perturbation.md``. -RED phase: these tests describe the public surface before it exists. +These tests exercise the public surface of the Paraphrase family. """ import pytest