Chart the two findings a table hides

A benchmark repository with no picture asked every reader to reconstruct the
result from twelve rows.

ndcg-intervals plots each configuration with its 95% interval and highlights
the three the paired bootstrap cannot separate from the best. Emphasis - one
hue plus grey - rather than a colour per row, because the reader's question is
"which of these actually differ", and that is a two-group question. Seeing the
intervals overlap does more than the table's ordering ever did.

abstention plots coverage against selective accuracy for both confidence
signals, which is the least expected result here: thresholding on the top-1
margin is worse than answering everything (0.25 against 0.47), because RRF
fuses as 1/(60+rank) and the top-two gap is ~2% on every query. Points resting
on fewer than five answered queries are dropped; one question should not draw
a cliff.

Both render in light and dark so the README can pair them with <picture>; a
light PNG on a dark page is a glare rectangle. Palette values come from a
validated reference and were re-checked on both surfaces for contrast and
colour-vision separation before use.

Title and subtitle are offset in points from the axes corner, not in axes
fractions: the interval chart's height grows with the number of rows, so a
fractional offset drifts and a fixed title pad collides.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EJ4G5AQVGTMMDin5XGvqoU

Author: RizgarOzan

.github/workflows/validate.yml

@@ -14,10 +14,10 @@ jobs:
       - uses: actions/setup-python@v5
         with:
           python-version: "3.12"
-      # The base install plus dev extras: enough for the metrics, the gold-set
-      # checks and the Turkish tokeniser. Dense retrieval pulls torch and is
-      # not exercised here, so it stays out.
-      - run: pip install -e '.[dev]'
+      # Base install plus dev and charts: enough for the metrics, the gold-set
+      # checks, the Turkish tokeniser and chart rendering. Dense retrieval pulls
+      # torch and is not exercised here, so it stays out.
+      - run: pip install -e '.[dev,charts]'
       - run: python -m pytest -q
       - name: Offline checks on every question file
         run: turkish-rag-eval validate

docs/charts/abstention-dark.png

@@ -0,0 +1,251 @@
+"""Render the two charts the README leads with.
+
+A benchmark repository with no picture in it asks every reader to reconstruct
+the finding from a twelve-row table. These two carry the findings that a table
+hides:
+
+``ndcg-intervals``  every configuration with its 95% bootstrap interval, with
+                    the ones the data cannot separate from the best picked out.
+                    The interval is the point: the ranking looks decisive and
+                    mostly is not.
+
+``abstention``      coverage against selective accuracy for both confidence
+                    signals. This is the repository's least expected result -
+                    the intuitive signal (top-1 margin) is useless on fused
+                    rankings, because RRF scores are 1/(60+rank) and the
+                    top-two gap is about 2% for every query, confident or not.
+
+Both are written in light and dark variants, because a light-background PNG in
+a dark README is a glare rectangle; the README pairs them with <picture>.
+
+Colours come from the reference palette and were validated for contrast and
+colour-vision separation on both surfaces before use. The emphasis chart is
+one hue plus grey rather than a colour per bar: the reader's question is
+"which of these are actually different", and that is a two-group question.
+"""
+
+import argparse
+import json
+from pathlib import Path
+
+from .paths import ROOT
+from .report import DEFAULT_METRIC, load_configurations, recommend
+
+#: Points resting on a handful of answered queries swing between 0 and 1 on
+#: one question. abstain.pick_threshold ignores them for the same reason.
+MIN_ANSWERED = 5
+
+THEMES = {
+    "light": {
+        "surface": "#fcfcfb",
+        "ink": "#0b0b0b",
+        "secondary": "#52514e",
+        "muted": "#898781",
+        "grid": "#e1e0d9",
+        "axis": "#c3c2b7",
+        "accent": "#2a78d6",
+        "second": "#eb6834",
+        "recessive": "#898781",
+    },
+    "dark": {
+        "surface": "#1a1a19",
+        "ink": "#ffffff",
+        "secondary": "#c3c2b7",
+        "muted": "#898781",
+        "grid": "#2c2c2a",
+        "axis": "#383835",
+        "accent": "#3987e5",
+        "second": "#d95926",
+        "recessive": "#898781",
+    },
+}
+
+
+def _figure(theme: dict, size: tuple[float, float]):
+    import matplotlib.pyplot as plt
+
+    figure, axes = plt.subplots(figsize=size)
+    figure.patch.set_facecolor(theme["surface"])
+    axes.set_facecolor(theme["surface"])
+    for side in ("top", "right"):
+        axes.spines[side].set_visible(False)
+    for side in ("left", "bottom"):
+        axes.spines[side].set_color(theme["axis"])
+        axes.spines[side].set_linewidth(1)
+    axes.tick_params(colors=theme["muted"], labelsize=9, length=0)
+    return figure, axes
+
+
+def _titles(axes, theme: dict, title: str, subtitle: str) -> None:
+    """Title and subtitle stacked above the axes.
+
+    Offset in *points* from the axes corner rather than in axes fractions: the
+    interval chart's height grows with the number of configurations, so a
+    fractional offset would drift away from the title as rows are added, and a
+    fixed title pad would collide with it.
+    """
+    subtitle_lines = subtitle.count("\n") + 1
+    axes.annotate(subtitle, xy=(0, 1), xycoords="axes fraction",
+                  xytext=(0, 10), textcoords="offset points",
+                  ha="left", va="bottom", fontsize=9.5,
+                  color=theme["secondary"], linespacing=1.45)
+    axes.annotate(title, xy=(0, 1), xycoords="axes fraction",
+                  xytext=(0, 10 + 15 * subtitle_lines + 8),
+                  textcoords="offset points", ha="left", va="bottom",
+                  fontsize=13.5, fontweight="bold", color=theme["ink"])
+
+
+def intervals_chart(results_dir: Path, out: Path, theme_name: str,
+                    metric: str = DEFAULT_METRIC) -> Path:
+    """Every configuration, sorted, with its interval; the tied ones picked out."""
+    theme = THEMES[theme_name]
+    configurations = load_configurations(results_dir)
+    recommendation = recommend(configurations, metric)
+    tied = {id(c) for c in recommendation.tied_with_best}
+
+    rows = sorted(configurations, key=lambda c: c.mean(metric))
+    figure, axes = _figure(theme, (8.4, 0.44 * len(rows) + 2.3))
+
+    for position, configuration in enumerate(rows):
+        low, high = configuration.interval(metric)
+        mean = configuration.mean(metric)
+        emphasised = id(configuration) in tied
+        colour = theme["accent"] if emphasised else theme["recessive"]
+
+        axes.plot([low, high], [position, position], color=colour, linewidth=2,
+                  solid_capstyle="round", zorder=2,
+                  alpha=1.0 if emphasised else 0.55)
+        axes.plot([mean], [position], marker="o", markersize=8, color=colour,
+                  markeredgecolor=theme["surface"], markeredgewidth=2, zorder=3,
+                  alpha=1.0 if emphasised else 0.55)
+        axes.text(high + 0.012, position, f"{mean:.3f}", va="center",
+                  fontsize=9, color=theme["ink"] if emphasised
+                  else theme["muted"])
+
+    axes.set_yticks(range(len(rows)))
+    axes.set_yticklabels([c.name for c in rows], fontsize=9.5,
+                         color=theme["secondary"])
+    axes.set_xlabel(metric, color=theme["secondary"], fontsize=9.5, labelpad=8)
+    axes.set_xlim(0.22, 0.82)
+    axes.set_ylim(-0.7, len(rows) - 0.3)
+    axes.xaxis.grid(True, color=theme["grid"], linewidth=1)
+    axes.set_axisbelow(True)
+
+    _titles(axes, theme,
+            "Most of this ranking is not a ranking",
+            f"{metric} with 95% bootstrap intervals over "
+            f"{len(recommendation.best.scored)} queries. Highlighted: the "
+            f"configurations a paired\nbootstrap cannot separate from the best.")
+
+    # A two-group emphasis chart still needs its groups named.
+    from matplotlib.lines import Line2D
+    axes.legend(
+        handles=[
+            Line2D([], [], color=theme["accent"], linewidth=2, marker="o",
+                   markersize=7, label="within noise of the best"),
+            Line2D([], [], color=theme["recessive"], alpha=0.55, linewidth=2,
+                   marker="o", markersize=7, label="measurably worse"),
+        ],
+        loc="lower right", frameon=False, fontsize=9,
+        labelcolor=theme["secondary"])
+
+    return _save(figure, out)
+
+
+def abstention_chart(results_dir: Path, out: Path, theme_name: str) -> Path:
+    """Coverage against selective accuracy, one line per confidence signal."""
+    theme = THEMES[theme_name]
+    figure, axes = _figure(theme, (7.6, 5.0))
+
+    series = [
+        ("score", "dense top-1 cosine", theme["accent"]),
+        ("margin", "top-1 margin", theme["second"]),
+    ]
+    plotted = 0
+    for signal, label, colour in series:
+        path = results_dir / f"abstain_curve_{signal}.json"
+        if not path.exists():
+            continue
+        curve = [p for p in json.loads(path.read_text(encoding="utf-8"))
+                 if p["answered"] >= MIN_ANSWERED
+                 and p["selective_accuracy"] is not None]
+        if not curve:
+            continue
+        coverage = [p["coverage"] for p in curve]
+        accuracy = [p["selective_accuracy"] for p in curve]
+
+        axes.plot(coverage, accuracy, color=colour, linewidth=2, marker="o",
+                  markersize=6, markeredgecolor=theme["surface"],
+                  markeredgewidth=1.5, label=label, zorder=3)
+        # Direct label at the low-coverage end, where the two lines separate.
+        leftmost = min(range(len(coverage)), key=lambda i: coverage[i])
+        axes.annotate(label, (coverage[leftmost], accuracy[leftmost]),
+                      textcoords="offset points", xytext=(8, 8),
+                      fontsize=9.5, color=theme["ink"], fontweight="bold")
+        plotted += 1
+
+    axes.set_xlabel("coverage - share of queries answered automatically",
+                    color=theme["secondary"], fontsize=9.5, labelpad=8)
+    axes.set_ylabel("selective accuracy - correct among those answered",
+                    color=theme["secondary"], fontsize=9.5, labelpad=8)
+    axes.set_xlim(0, 1.03)
+    axes.set_ylim(0, 1.03)
+    axes.grid(True, color=theme["grid"], linewidth=1)
+    axes.set_axisbelow(True)
+    if plotted > 1:
+        axes.legend(loc="lower left", frameon=False, fontsize=9,
+                    labelcolor=theme["secondary"])
+
+    _titles(axes, theme,
+            "The intuitive confidence signal is the useless one",
+            f"Thresholds keeping at least {MIN_ANSWERED} answered queries. RRF "
+            f"fuses ranks as 1/(60+rank), so the top-two\ngap is ~2% on every "
+            f"query and the margin buys almost no coverage to trade.")
+    return _save(figure, out)
+
+
+def _save(figure, out: Path) -> Path:
+    import matplotlib.pyplot as plt
+
+    out.parent.mkdir(parents=True, exist_ok=True)
+    figure.savefig(out, dpi=200, bbox_inches="tight",
+                   facecolor=figure.get_facecolor())
+    plt.close(figure)
+    return out
+
+
+def add_arguments(parser: argparse.ArgumentParser) -> None:
+    parser.add_argument("--results", type=Path, default=ROOT / "results")
+    parser.add_argument("--out", type=Path, default=ROOT / "docs" / "charts")
+    parser.add_argument("--metric", default=DEFAULT_METRIC)
+
+
+def run(args) -> int:
+    if not (args.results / "summary.json").exists():
+        print(f"{args.results} holds no results - run 'turkish-rag-eval run' first")
+        return 1
+
+    written = []
+    for theme_name in THEMES:
+        suffix = "" if theme_name == "light" else "-dark"
+        written.append(intervals_chart(
+            args.results, args.out / f"ndcg-intervals{suffix}.png",
+            theme_name, args.metric))
+        if (args.results / "abstain_curve_score.json").exists():
+            written.append(abstention_chart(
+                args.results, args.out / f"abstention{suffix}.png", theme_name))
+
+    for path in written:
+        print(f"  {path.relative_to(ROOT) if ROOT in path.parents else path}")
+    print(f"{len(written)} chart(s) written")
+    return 0
+
+
+def main() -> int:
+    parser = argparse.ArgumentParser(description=__doc__)
+    add_arguments(parser)
+    return run(parser.parse_args())
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

docs/charts/abstention.png

@@ -32,6 +32,7 @@
     "report": ("report", "turn results into a recommendation"),
     "abstain": ("run_abstain", "produce abstain records and the coverage curve"),
     "abstain-report": ("abstain", "re-print the coverage curve from saved records"),
+    "charts": ("charts", "render the result charts the README embeds"),
     "agreement": ("agreement", "inter-annotator agreement over the gold set"),
     "fetch-corpus": ("fetch_corpus", "download the Wikipedia corpus snapshot"),
     "export-hf": ("export_hf", "export the BEIR layout MTEB reads"),

docs/charts/ndcg-intervals-dark.png

@@ -0,0 +1,98 @@
+"""Charts are checked for the things a rendered image cannot tell you.
+
+Whether they look right is settled by opening them. What is worth pinning is
+that both themes are produced, that a results directory without the abstention
+run still yields the chart that does exist, and that the noisiest end of the
+coverage curve is filtered out rather than plotted.
+"""
+
+import json
+
+import pytest
+
+pytest.importorskip("matplotlib")
+pytest.importorskip("numpy")
+
+from turkish_rag_eval import charts  # noqa: E402
+
+
+def records(hits):
+    return [{"question": f"q{i}", "relevance": [hit], "total_relevant": 1,
+             "found": hit} for i, hit in enumerate(hits)]
+
+
+@pytest.fixture
+def results(tmp_path):
+    directory = tmp_path / "results"
+    directory.mkdir()
+    for name, hits in [("sentence_dense", [True, False] * 15),
+                       ("fixed_bm25_stem5", [True, True, False] * 10)]:
+        (directory / f"perquery_{name}.json").write_text(
+            json.dumps(records(hits)), encoding="utf-8")
+    (directory / "summary.json").write_text(json.dumps([
+        {"chunking": "sentence", "retriever": "dense", "latency_ms_p95": 19},
+        {"chunking": "fixed", "retriever": "bm25_stem5", "latency_ms_p95": 4},
+    ]), encoding="utf-8")
+    return directory
+
+
+def curve(points):
+    return [{"threshold": t, "coverage": c, "selective_accuracy": a,
+             "answered": n, "escalated": 58 - n} for t, c, a, n in points]
+
+
+def test_both_themes_are_written(results, tmp_path):
+    for theme in ("light", "dark"):
+        out = tmp_path / f"{theme}.png"
+        charts.intervals_chart(results, out, theme)
+        assert out.stat().st_size > 5000, "a blank PNG would be much smaller"
+
+
+def test_unknown_theme_is_rejected(results, tmp_path):
+    with pytest.raises(KeyError):
+        charts.intervals_chart(results, tmp_path / "x.png", "solarized")
+
+
+def test_thin_thresholds_are_left_out(results, tmp_path):
+    # One answered query puts a point at exactly 0.0 or 1.0 accuracy; plotting
+    # it would draw a cliff that is one question wide.
+    (results / "abstain_curve_score.json").write_text(json.dumps(curve([
+        (0.0, 1.0, 0.47, 58),
+        (0.5, 0.40, 0.70, 23),
+        (0.9, 0.02, 1.00, 1),
+    ])), encoding="utf-8")
+
+    out = charts.abstention_chart(results, tmp_path / "abstention.png", "light")
+    assert out.exists()
+
+
+def test_a_curve_of_only_thin_points_draws_no_line(results, tmp_path):
+    (results / "abstain_curve_score.json").write_text(
+        json.dumps(curve([(0.9, 0.02, 1.0, 1)])), encoding="utf-8")
+    # Still produces a labelled, empty chart rather than raising.
+    assert charts.abstention_chart(results, tmp_path / "empty.png", "light").exists()
+
+
+def test_run_without_results_explains_itself(tmp_path, capsys):
+    import argparse
+
+    parser = argparse.ArgumentParser()
+    charts.add_arguments(parser)
+    args = parser.parse_args([])
+    args.results, args.out = tmp_path, tmp_path / "out"
+
+    assert charts.run(args) == 1
+    assert "turkish-rag-eval run" in capsys.readouterr().out
+
+
+def test_run_writes_charts_for_every_theme(results, tmp_path):
+    import argparse
+
+    parser = argparse.ArgumentParser()
+    charts.add_arguments(parser)
+    args = parser.parse_args([])
+    args.results, args.out = results, tmp_path / "out"
+
+    assert charts.run(args) == 0
+    written = sorted(p.name for p in (tmp_path / "out").glob("*.png"))
+    assert written == ["ndcg-intervals-dark.png", "ndcg-intervals.png"]