case-studies

Case studies

Worked tours of FalsifyAI's evidence infrastructure over real preserved artifacts. Each case study is itself a FalsifyAI artifact: a ReplayStore bundle (or, for 08, a committed reproducible spec) plus prose that walks through what history, diff, inspect, and replay reveal when read against it.

Recursion principle: claims about a stochastic system's reliability are only defensible if the evidence supporting them is preserved and replayable. The case studies below are written in FalsifyAI's own format — every command shown reproduces the displayed output verbatim against a bundled SQLite store (08 is the exception: minimize is an orchestrator, so it reproduces live from the committed spec).

For the categorical framing the case studies operationalize — why capability scores and reliability evidence answer different questions — see ../THE-EVIDENCE-GAP.md.

The arc. Read in order, the case studies widen the question the preserved evidence answers. It can show a regression — a contract breaks under perturbation and the artifact catches it (01, 07). It can explain one — inspect / replay open the run to show why, whether the fault is a real model contradiction (04) or, three times over, FalsifyAI's own machinery misfiring (03 / 05 / 06, the self-falsification trilogy spanning the interpretation and generation layers). It can track drift across sessions, so change is a comparison rather than a memory (history / diff / --show-timeline; 01, 02). And — newest — it can measure how much pressure a contract tolerates before it breaks (08, via minimize). The series moves from did it break? to how fragile, exactly? — each step the same evidence substrate read by a different tool.

Index

#	Title	What it demonstrates	Tools used
01	Invisible character substitution	Cross-model contains-contract brittleness as a persistent class; a model-migration regression (U+202F substitution between "30" and "days") as the vivid instance.	history, diff, inspect, replay
02	Resolver arbitration: boundary shift without verdict shift	An operationally motivated CLAUDE.md revision changed where a model permitted additional architectural complexity to exist without changing its top-level recommendation — the kind of subtle drift a pass/fail evaluator would miss. Manual retrospective probe; reproduction specs at specs/.	manual probe + machine-reproducible specs/
03	When the evaluator is wrong: a self-falsification study	A correct, stable model assigned the framework's harshest verdict (CONSISTENTLY_WRONG @ 1.00) by FalsifyAI's own interpretation layer; the preserved evidence overturned every false-positive verdict and drove the 0.6.1 HallucinationOracle fix. Self-falsification — differs from the model-behavior charter on purpose.	inspect over the bundled probe-03/ store
04	Overconfident negation: when CONSISTENTLY_WRONG means it	The companion to 03. A downgraded model (llama-3.1-8b-instant) reads a retention clause, cites the legal carve-out, then stably answers the wrong yes/no — a genuine CONSISTENTLY_WRONG. Validates the 0.6.1 oracle fix in the other direction: it still fires on real contradiction (true positive) after it stopped firing on NEUTRAL (false positive).	inspect over the bundled store
05	When the confidence number lies: a presentation-layer self-falsification	A second reading of the same probe-03/ bundle: instability-band verdicts (ADVERSARIALLY_VULNERABLE / FRAGILE / AMBIGUOUS) rendered confidence: 0.00 — a number that inverts (the stability floor reads as low certainty when it signals high severity). Orthogonal to 03: verdict and value are correct, only the label was wrong. Drove a band-aware consumer-surface fix; resolver byte-identical.	replay over the bundled store
06	When the test deletes the question: a generation-layer self-falsification	The third self-falsification, on the layer 03 and 05 left untouched: a paraphrase rewrite deleted the task's grounding and passed the cosine validity gate (topically similar, semantically gutted), manufacturing CONSISTENTLY_WRONG over a correct llama-3.1-8b-instant. Drove the §9.3 BidirectionalNLIValidator — entailment, both directions — that the MVP had deferred. Generation layer; resolver untouched.	inspect over the bundled probe-06/ before/after stores
07	The regression that only appeared under pressure	The outward-facing adoption study: a model migration (llama-3.3-70b → cheaper llama-3.1-8b) passed the clean eval on both, then under realistic input noise the cheaper model dropped a customer-facing contract clause (a promo's gift-card exclusion) from one summary. diff flagged it (exit 5), inspect showed the dropped clause, verify + export made the proof portable. Not a self-falsification — the everyday migration-safety job.	diff, inspect, verify, export over the bundled data/case-study-07.db
08	How fragile, exactly? The minimal falsifier	The quantification companion to 07: minimize reports the minimal falsifier — the smallest typo_noise strength that flips a case out of STABLE. On 07's promo contract the 8B candidate first leaves STABLE at 0.05 (3/3 runs) while the 70B holds through 0.1 and breaks at 0.2–0.4 — a conservative 4× threshold gap a single fixed-pressure test cannot see. No frozen bundle: minimize prints rather than writing a store, so this is reproducible from the committed spec like 02.	minimize (reproducible from probe-07/ specs, no bundle)

How to read a case study

Each case study has the same shape:

1. Setup — what was run, against what models, with what spec.
2. A systemic finding (the thesis) — what history shows across multiple sessions.
3. A specific instance (the vivid proof) — what diff and inspect reveal in one session; typically 1–3 sub-sections detailing the regression, the evidence, and the failure mechanism.
4. Why it matters — the operational point that makes the evidence load-bearing (e.g. why semantic matching wouldn't catch this).
5. Reproduction — exact commands the reader runs against the bundled store.
6. Synthesis — the architectural claim the case study exists to demonstrate (typically: one preserved evidence substrate, multiple consumer surfaces).

The bundle lives in data/ and has a provenance README recording SHA256, environment, and session-to-model mappings.

What case studies are NOT

• Not benchmarks or leaderboards (no model ranking).
• Not marketing posts or launch announcements.
• Not tutorials (tutorials teach how-to; case studies preserve what-happened).
• Not critiques of any model or provider — they document reliability-contract behavior under perturbation, not model quality.
• Not synthesized data. The artifacts bundled here are real runs from real campaigns, copied verbatim.

Adding a new case study

The pattern that makes a case study load-bearing:

1. The evidence already exists in a ReplayStore from a real run — never synthesize.
2. There is a systemic finding and a specific instance — neither alone is strong enough.
3. The bundle is included so every command shown is verifiable.
4. The prose passes the framing test: would a careful reader interpret this as reliability pressure testing or as adversarial gotcha? Only the former is acceptable.
5. The synthesis names the architectural claim the case study exists to demonstrate.

If you have a real run that meets all five, propose a case study by opening an issue with the session ID(s) and the architectural claim you'd be demonstrating.