Publish recorded benchmark results, and name the OCR model and tokens for every run

.github/workflows/benchmark-pages.yml

@@ -48,12 +48,18 @@ jobs:
           mkdir -p .benchmark
           uv run pytest tests/integration --junitxml=.benchmark/integration.xml
 
+      # The conversions are replayed from site/snapshot.json rather than run here.
+      # A runner has no vision backend, so a live build could only ever measure
+      # the no-OCR configuration, and the full corpus takes far longer than a
+      # Pages build should. The snapshot is recorded by a maintainer with all
+      # three backends available; see site/README.md for how to refresh it.
       - name: Generate Pages artifact
         run: |
           uv run python site/generate.py \
             --output _site \
             --junit .benchmark/integration.xml \
             --skip-synthetic \
+            --snapshot site/snapshot.json \
             --commit "${GITHUB_SHA}" \
             --ref "${GITHUB_REF_NAME}"
 
@@ -63,6 +69,12 @@ jobs:
         run: |
           uv run python -c "import json,sys; data=json.load(open('_site/data/results.json')); sys.exit(data['summary']['fail'] > 0)"
 
+      # Put the published numbers in the job summary so a regression is visible
+      # from the Actions run itself, without deploying or downloading the artifact.
+      - name: Publish benchmark numbers
+        if: always()
+        run: uv run python site/summarize.py _site/data/results.json >> "${GITHUB_STEP_SUMMARY}"
+
       - name: Upload Pages artifact
         uses: actions/upload-pages-artifact@56afc609e74202658d3ffba0e8f6dda462b719fa # v3
         with:

README.md

@@ -16,6 +16,9 @@ pip install likhit
 
 - Website: https://jawafdehi.org/
 - GitHub: https://github.com/Jawafdehi/likhit/
+- Benchmark: https://jawafdehi.github.io/likhit/ — real, published Government of
+  Nepal documents converted with and without OCR, with the extracted Markdown and
+  the assertions behind every result
 - Contact: inquiry@jawafdehi.org
 
 ## Usage
@@ -134,6 +137,57 @@ Optional variables:
 export MARKITDOWN_OCR_PROMPT="Custom OCR instructions"
 ```
 
+#### Recipe: a vision model on your own machine
+
+Ollama, llama.cpp and vLLM all serve an OpenAI-compatible endpoint, so no `likhit`
+change is needed — and no page image leaves the machine:
+
+```bash
+ollama pull qwen2.5vl:7b
+
+# Ollama ignores the key; the client refuses to start without one
+export OPENAI_API_KEY="ollama"
+export OPENAI_BASE_URL="http://127.0.0.1:11434/v1"
+export MARKITDOWN_OCR_MODEL="qwen2.5vl:7b"
+
+likhit-save scanned-notice.pdf --out notice.md
+```
+
+#### Recipe: AWS Bedrock, through a gateway
+
+Bedrock does not speak the OpenAI API, so put a translating proxy in front of it.
+The proxy resolves AWS credentials the usual way — `AWS_PROFILE`, environment
+variables or an instance role — and `likhit` only ever sees an OpenAI endpoint:
+
+```bash
+pip install 'litellm[proxy]'
+litellm --model bedrock/anthropic.claude-sonnet-5
+
+# whatever you configured the proxy to accept
+export OPENAI_API_KEY="local-proxy-key"
+export OPENAI_BASE_URL="http://127.0.0.1:4000/v1"
+export MARKITDOWN_OCR_MODEL="bedrock/anthropic.claude-sonnet-5"
+```
+
+Any vision-capable model your account has enabled will do; swap the model id for
+one you have access to.
+
+#### When OCR actually runs
+
+`likhit` adds an OCR candidate only for pages whose text layer cannot serve them
+(see [Architecture](#architecture) below), so configuring a vision backend does
+not mean paying for one on every document. In the published benchmark, 13 of 16
+Government of Nepal documents are converted without a single vision call; only the
+image-only scans need one.
+
+If OCR is left unconfigured, `likhit` still converts. It logs that a page needed
+OCR and returns the best result it could reach without it, rather than failing:
+
+```text
+PDF converter: OCR appears necessary, but OCR is not configured.
+Set OPENAI_API_KEY or GEMINI_API_KEY, plus MARKITDOWN_OCR_MODEL, to enable markitdown-ocr.
+```
+
 ## Architecture
 
 The high-level PDF pipeline is:

site/README.md

@@ -32,8 +32,11 @@ Every document runs against three configurations:
 
 A configuration declares the backend it `requires`, and one whose backend is
 absent is **skipped rather than run** — otherwise a missing credential would be
-recorded as a Likhit defect. CI supplies neither backend, so the published build
-runs `likhit` alone and reports the other two as unavailable with a reason.
+recorded as a Likhit defect. That is what happens on a live build with no
+credentials: `likhit` runs alone and the other two are reported unavailable with a
+reason. The deployed site is not a live build — it replays a recording and
+publishes all three columns, taking availability from the record rather than from
+the runner; see [Recorded results](#recorded-results-snapshotjson).
 
 Point the hosted backend at a provider with `MARKITDOWN_OCR_MODEL` plus
 `OPENAI_API_KEY` (or `GEMINI_API_KEY`); see the root README for the full OCR
@@ -51,11 +54,114 @@ configuration may raise its own `timeout_s` in `catalog.json`, which the offline
 backend does: vision inference on CPU is far slower than a hosted API, and
 timing it out would misreport a slow machine as a conversion failure.
 
-If a backend exposes a cumulative token counter, set `LIKHIT_OCR_USAGE_URL` (or
-`LIKHIT_LOCAL_OCR_USAGE_URL`) and each run records the calls and tokens it spent,
-along with the model that spent them. Tokens only — no cost is derived, because
-vendor rates are not published for every model and a built-in price table would
-silently produce wrong money.
+## Token usage
+
+Nothing in Likhit or MarkItDown exposes token usage — the figures are in the OCR
+response, which the converter does not surface — so the generator reads a
+cumulative counter and takes the difference across each conversion. Point
+`LIKHIT_OCR_USAGE_URL` (or `LIKHIT_LOCAL_OCR_USAGE_URL`) at one that serves
+`{"calls", "input_tokens", "output_tokens"}`.
+
+`ocr_usage_proxy.py` is a reference implementation: it forwards to any
+OpenAI-compatible endpoint and accumulates the `usage` block each response
+carries. Run one per backend, so a hosted run can never be credited with a
+locally served model's spend:
+
+```bash
+python site/ocr_usage_proxy.py --port 8141 --upstream http://127.0.0.1:11434
+export LIKHIT_LOCAL_OCR_BASE_URL=http://127.0.0.1:8141/v1
+export LIKHIT_LOCAL_OCR_USAGE_URL=http://127.0.0.1:8141/usage
+```
+
+**Zero is a measurement, not a gap.** Likhit adds an OCR candidate only for pages
+a text layer cannot serve, so most documents spend nothing even with a vision
+backend configured — in the committed recording, 13 of 16 hosted runs make no
+call at all. Those record `calls: 0` and the dashboard says "no OCR call";
+`ocr_usage: null` is reserved for a counter that was genuinely unreachable, and
+renders as "not recorded". Collapsing the two would make the common case
+indistinguishable from lost data.
+
+The model id is recorded per **configuration**, not per run, because a run that
+made no call has no usage record to carry it — and "which model produced this
+column" is exactly what the dashboard needs to answer. It is read from the
+environment when recording and from the snapshot when replaying.
+
+Tokens only — no cost is derived, because vendor rates are not published for
+every model and a built-in price table would silently produce wrong money.
+
+## Recorded results (`snapshot.json`)
+
+The deployed site does not measure anything. CI has no vision backend, and
+converting the whole corpus three ways takes far longer than a Pages build, so
+`--snapshot site/snapshot.json` **replays a recorded run** instead:
+
+```bash
+uv run python site/generate.py --output _site --skip-synthetic \
+  --snapshot site/snapshot.json
+```
+
+A replay performs no conversion and calls no backend. It takes each run's status,
+text, timing, memory and OCR usage from the record, and takes **availability from
+the record too** — otherwise CI would skip the very OCR columns the snapshot
+exists to publish. Everything downstream is recomputed from the recorded text, so
+editing a check threshold in `catalog.json` changes a replayed result without
+re-recording. Sources are still downloaded and hash-checked, so a replayed build
+still needs network access to the publisher URLs; only the conversions are
+replayed.
+
+Re-record after any change to conversion behaviour, **from a machine where all
+three backends work** — the recording is only as complete as the environment that
+produced it:
+
+```bash
+uv run python site/generate.py --output _site --skip-synthetic \
+  --write-snapshot site/snapshot.json
+```
+
+The two flags are mutually exclusive — `generate()` rejects the pair, not just the
+CLI — because reading a recording while writing one would copy it forward under a
+new timestamp without measuring anything, stamping the replaying build's commit
+onto numbers it never took. `--write-snapshot` prints the configurations it
+covered; check that line, because a snapshot recorded without a backend records
+that configuration as unavailable, and the published page then shows it as not
+run. `test_committed_snapshot_covers_every_published_run_and_configuration` fails
+when the committed file stops covering the catalog.
+
+Two unavailable configurations are **not** the same thing, and the artifact keeps
+them apart. One the recording covers as `available: false` had no backend when the
+recording was made, and carries that reason. One the recording never mentions —
+because the catalog gained it afterwards — has no measurement at all: its runs are
+named in `measured.missing_runs`, and its reason is "not covered by the recording"
+rather than a claim about credentials that says nothing about the gap.
+
+The committed snapshot holds the extracted text of every run (~700 KB). That is
+the evidence behind the numbers, and it is the same text already published to
+Pages, so committing it exposes nothing new.
+
+Provenance is deliberately split in the artifact: `build` is the commit that
+published the page, `measured` is the commit whose behaviour the numbers describe.
+`measured.stale` marks the case where they differ and `measured.missing_runs` names
+catalog runs the recording predates; the dashboard renders both above the summary.
+A live build sets `measured` to null.
+
+`stale` is reported, not warned about. Committing a recording necessarily creates a
+commit later than the one it was recorded on, so it is true on effectively every
+deploy — warning on it would fire every time and teach readers to ignore the
+banner. The recorded commit and the published one are both named and left at that.
+`missing_runs` is the opposite case: a real gap, which does warn.
+
+## Job summary
+
+`summarize.py` renders an artifact as Markdown for `$GITHUB_STEP_SUMMARY`, so the
+published numbers, the per-configuration outcomes and any missing-run warning
+appear on the Actions run itself rather than only in the deployed page:
+
+```bash
+uv run python site/summarize.py _site/data/results.json
+```
+
+It runs with `if: always()` and exits 0 when the artifact is absent, so a failed
+generation is not replaced by a second, less useful error.
 
 ## Artifact contract
 

site/catalog.json

@@ -909,7 +909,8 @@
         {
           "id": "likhit-ocr-local",
           "config": "likhit-ocr-local",
-          "expectation": "pass",
+          "comment": "Intermittently slow, not reliably broken: the locally served model exhausted the 2400s budget on one recording of this document and completed the next in 1073s, against about a minute for the hosted backend. Marked known_issue so a timeout is reported rather than failing the build, while a run that completes and meets the check is promoted straight back to pass -- which is what the current recording shows. Revisit if it times out repeatedly.",
+          "expectation": "known_issue",
           "checks": [
             {
               "kind": "min_devanagari",