diff --git a/bench/REPORT_MATRIX.md b/bench/REPORT_MATRIX.md index 8e04e75..edfee08 100644 --- a/bench/REPORT_MATRIX.md +++ b/bench/REPORT_MATRIX.md @@ -1,26 +1,26 @@ # ๐Ÿ›ก๏ธ anonde bench matrix -**`anonde-ner` โ€” the default NER image โ€” is the lowest-leak PII redactor in this benchmark: it misses just 11.1% of gold PII vs Presidio 41.8% / raw GLiNER 33.1% / OpenAI Privacy Filter 24.0%, across 29 gold-annotated corpora and 5 languages.** Tuned recall-first โ€” it catches more PII than the precision-optimised tools, at the cost of more over-redaction (quantified two lines down). +**`anonde-ner` โ€” the default NER image โ€” is the lowest-leak PII redactor in this benchmark: it misses just 11.1% of gold PII vs Presidio 41.7% / raw GLiNER 33.1% / OpenAI Privacy Filter 24.0%, across 29 gold-annotated corpora and 5 languages.** Tuned recall-first โ€” it catches more PII than the precision-optimised tools, at the cost of more over-redaction (quantified two lines down). -| Language | `anonde-ner` โฌ… ours | `presidio` | `presidio-transformer` | `gliner-py` | `openai-pf` | -|---|---:|---:|---:|---:|---:| -| **English** | **9.8%** ๐Ÿฅ‡ | 36.5% | 38.1% | 39.8% | 19.5% | -| **German** | **6.4%** ๐Ÿฅ‡ | 38.3% | โ€“ | 31.9% | 28.8% | -| **Spanish** | **15.8%** ๐Ÿฅ‡ | 47.0% | โ€“ | 29.3% | 25.7% | -| **French** | **13.9%** ๐Ÿฅ‡ | 43.1% | โ€“ | 30.7% | 21.1% | -| **Italian** | **14.6%** ๐Ÿฅ‡ | 48.5% | โ€“ | 33.8% | 21.0% | -| **All** | **11.1%** ๐Ÿฅ‡ | **41.8%** | **38.1%** | **33.1%** | **24.0%** | +| Language | `anonde-ner` โฌ… ours | `presidio` | `gliner-py` | `openai-pf` | +|---|---:|---:|---:|---:| +| **English** | **9.8%** ๐Ÿฅ‡ | 36.5% | 39.8% | 19.4% | +| **German** | **6.4%** ๐Ÿฅ‡ | 38.3% | 31.9% | 28.7% | +| **Spanish** | **15.8%** ๐Ÿฅ‡ | 47.0% | 29.3% | 25.8% | +| **French** | **13.9%** ๐Ÿฅ‡ | 43.1% | 30.7% | 21.1% | +| **Italian** | **14.6%** ๐Ÿฅ‡ | 48.5% | 33.8% | 21.0% | +| **All** | **11.1%** ๐Ÿฅ‡ | **41.7%** | **33.1%** | **24.0%** | *The one table. **Leak rate** = fraction of gold PII spans **missed** โ€” lower is better; ๐Ÿฅ‡ = lowest-leak engine in the row. Columns are the default NER image `anonde-ner` vs the competing field (anonde's own patterns / stack tiers and the per-domain roll-ups are under **Details**). `presidio-transformer` is EN-only (`โ€“` elsewhere, by design); `openai-pf` is scored on a fixed per-corpus subsample. Full method, precision, and every slice are in **Details** below.* -| Language | `anonde-ner` โฌ… ours | `presidio` | `presidio-transformer` | `gliner-py` | `openai-pf` | -|---|---:|---:|---:|---:|---:| -| **English** | **0.563** ๐Ÿฅ‡ | 0.404 | 0.464 | 0.441 | 0.002 | -| **German** | **0.595** ๐Ÿฅ‡ | 0.385 | โ€“ | 0.479 | 0.006 | -| **Spanish** | **0.555** ๐Ÿฅ‡ | 0.345 | โ€“ | 0.511 | 0.000 | -| **French** | **0.682** ๐Ÿฅ‡ | 0.340 | โ€“ | 0.536 | 0.003 | -| **Italian** | **0.633** ๐Ÿฅ‡ | 0.320 | โ€“ | 0.480 | 0.004 | -| **All** | **0.599** ๐Ÿฅ‡ | **0.365** | **0.464** | **0.487** | **0.003** | +| Language | `anonde-ner` โฌ… ours | `presidio` | `gliner-py` | `openai-pf` | +|---|---:|---:|---:|---:| +| **English** | **0.563** ๐Ÿฅ‡ | 0.404 | 0.441 | 0.002 | +| **German** | **0.596** ๐Ÿฅ‡ | 0.385 | 0.480 | 0.006 | +| **Spanish** | **0.555** ๐Ÿฅ‡ | 0.345 | 0.511 | 0.000 | +| **French** | **0.682** ๐Ÿฅ‡ | 0.340 | 0.536 | 0.003 | +| **Italian** | **0.633** ๐Ÿฅ‡ | 0.320 | 0.480 | 0.004 | +| **All** | **0.599** ๐Ÿฅ‡ | **0.365** | **0.487** | **0.003** | *The twin. **Strict F1** = exact span **and** type match (CoNLL) โ€” higher is better, ๐Ÿฅ‡ = best in row. It reproduces the standard scorer (`nervaluate`) *exactly* (ฮ”โ‰ˆ0 in `verify_official.py`), so it is the citable accuracy metric alongside leak rate. It is precision-inclusive, so it also reflects over-redaction: an over-redacting tool can rank lower here than on leak rate, where a precision-first rival edges ahead. The lenient overlap view and full method are under **Details** / `METHODOLOGY.md`.* @@ -38,53 +38,51 @@ The full working behind the scorecard above โ€” leak-rate and precision roll-ups The one table. Roll-up rows only (per domain ยท per language ยท overall); the per-(domain ร— language) detail grid lives in the Detailed breakdown below. Each number is **leak rate** (fraction of gold PHI spans missed โ€” lower is better). `anonde-ner` is the default NER image (`ghcr.io/anonde-io/anonde-ner`) and the anchor column; **Verdict** says whether it beats the field. ๐Ÿฅ‡ marks the lowest-leak engine in the row. Roll-up rows pool leaked-over-gold across the group (doc-weighted, so larger corpora count more). -| Slice | Scope | `anonde-ner` โฌ…๏ธŽ anonde (default NER) | `anonde-patterns` | `presidio` | `presidio-transformer` | `gliner-py` | `openai-pf` | Verdict | -|---|---|---:|---:|---:|---:|---:|---:|:--:| -| **ฮฃ ALL** | **all** | **11.1%** ๐Ÿฅ‡ | **46.1%** | **41.8%** | **38.1%** | **33.1%** | **24.0%** | โœ… | -| | | | | | | | | | -| _ฮฃ Clinical / medical de-identification_ | _all langs_ | **11.3%** ๐Ÿฅ‡ | 47.5% | 31.1% | 15.2% | 28.0% | 27.3% | โœ… | -| _ฮฃ Legal / administrative_ | _all langs_ | **2.8%** ๐Ÿฅ‡ | 23.1% | 29.6% | 13.3% | 21.5% | 36.8% | โœ… | -| _ฮฃ Retail finance_ | _all langs_ | **8.2%** ๐Ÿฅ‡ | 24.4% | 21.3% | 12.0% | 23.5% | 19.0% | โœ… | -| _ฮฃ Enterprise logs_ | _all langs_ | **13.2%** ๐Ÿฅ‡ | 28.9% | 31.5% | 37.3% | 73.2% | 15.3% | โœ… | -| _ฮฃ General structured PII_ | _all langs_ | **13.4%** ๐Ÿฅ‡ | 62.1% | 57.8% | 57.4% | 34.7% | 18.8% | โœ… | -| _ฮฃ Academic NER (newswire / social)_ | _all langs_ | **6.1%** ๐Ÿฅ‡ | 68.4% | 18.3% | 16.5% | 13.8% | 72.7% | โœ… | -| _ฮฃ Adversarial / out-of-distribution_ | _all langs_ | **8.1%** ๐Ÿฅ‡ | 12.7% | 37.6% | โ€“ | 43.4% | 32.0% | โœ… | -| | | | | | | | | | -| _ฮฃ all domains_ | _English_ | **9.8%** ๐Ÿฅ‡ | 35.6% | 36.5% | 38.1% | 39.8% | 19.5% | โœ… | -| _ฮฃ all domains_ | _German_ | **6.4%** ๐Ÿฅ‡ | 25.0% | 38.3% | โ€“ | 31.9% | 28.8% | โœ… | -| _ฮฃ all domains_ | _Spanish_ | **15.8%** ๐Ÿฅ‡ | 69.9% | 47.0% | โ€“ | 29.3% | 25.7% | โœ… | -| _ฮฃ all domains_ | _French_ | **13.9%** ๐Ÿฅ‡ | 63.6% | 43.1% | โ€“ | 30.7% | 21.1% | โœ… | -| _ฮฃ all domains_ | _Italian_ | **14.6%** ๐Ÿฅ‡ | 58.6% | 48.5% | โ€“ | 33.8% | 21.0% | โœ… | +| Slice | Scope | `anonde-ner` โฌ…๏ธŽ anonde (default NER) | `anonde-patterns` | `presidio` | `gliner-py` | `openai-pf` | Verdict | +|---|---|---:|---:|---:|---:|---:|:--:| +| **ฮฃ ALL** | **all** | **11.1%** ๐Ÿฅ‡ | **46.1%** | **41.7%** | **33.1%** | **24.0%** | โœ… | +| | | | | | | | | +| _ฮฃ Clinical / medical de-identification_ | _all langs_ | **11.3%** ๐Ÿฅ‡ | 47.5% | 31.1% | 28.0% | 27.3% | โœ… | +| _ฮฃ Legal / administrative_ | _all langs_ | **2.8%** ๐Ÿฅ‡ | 23.1% | 29.6% | 21.5% | 36.8% | โœ… | +| _ฮฃ Retail finance_ | _all langs_ | **8.2%** ๐Ÿฅ‡ | 24.4% | 21.3% | 23.5% | 18.9% | โœ… | +| _ฮฃ Enterprise logs_ | _all langs_ | **13.2%** ๐Ÿฅ‡ | 28.9% | 31.5% | 73.2% | 15.2% | โœ… | +| _ฮฃ General structured PII_ | _all langs_ | **13.4%** ๐Ÿฅ‡ | 62.1% | 57.8% | 34.7% | 18.8% | โœ… | +| _ฮฃ Academic NER (newswire / social)_ | _all langs_ | **6.1%** ๐Ÿฅ‡ | 68.4% | 18.3% | 13.8% | 72.7% | โœ… | +| _ฮฃ Adversarial / out-of-distribution_ | _all langs_ | **8.0%** ๐Ÿฅ‡ | 12.6% | 37.5% | 43.3% | 32.0% | โœ… | +| | | | | | | | | +| _ฮฃ all domains_ | _English_ | **9.8%** ๐Ÿฅ‡ | 35.6% | 36.5% | 39.8% | 19.4% | โœ… | +| _ฮฃ all domains_ | _German_ | **6.4%** ๐Ÿฅ‡ | 25.0% | 38.3% | 31.9% | 28.7% | โœ… | +| _ฮฃ all domains_ | _Spanish_ | **15.8%** ๐Ÿฅ‡ | 69.9% | 47.0% | 29.3% | 25.8% | โœ… | +| _ฮฃ all domains_ | _French_ | **13.9%** ๐Ÿฅ‡ | 63.6% | 43.1% | 30.7% | 21.1% | โœ… | +| _ฮฃ all domains_ | _Italian_ | **14.6%** ๐Ÿฅ‡ | 58.6% | 48.5% | 33.8% | 21.0% | โœ… | > **Anonde scoreboard** โ€” across the **24** populated `(domain, language)` cells in the matrix, `anonde-ner` is the **lowest-leak engine in 23**, ties in **0**, and is beaten in **1**. โœ… = anonde leads ยท ๐ŸŸฐ = tied ยท โŒ = a baseline leaks less. See the per-cell leak-rate grid in the Detailed breakdown below for which baseline wins where. (The TL;DR's win count is per-corpus, a finer split than these per-cell rows.) -> **EN-only column** โ€” `presidio-transformer`: English corpora only. `presidio-transformer` is Presidio's `en_core_web_trf` config (an English transformer model), benchmarked next to the default `en_core_web_lg` `presidio` column so the report shows both Presidio configs on English. Non-EN cells render `โ€“` **by design** โ€” not a failed run โ€” and the roll-up rows pool leak rate over the English corpora only (partial coverage, like a subsampled engine). Compare against the other engines on the English rows only. - ## ๐ŸŽฏ Precision scorecard ยท false-positive rate roll-ups Partial precision = fraction of redacted spans that overlap a real PII span; the inverse (**1 โˆ’ precision**) is the over-redaction / false-positive rate. **Higher is better.** This is the overlap-based *partial* view (a predicted span counts as a true positive if it overlaps **any** gold span), NOT the strict byte-exact view โ€” strict punishes a one-char offset as a full false positive and reads misleadingly low (~0.1โ€“0.3) for every engine including the baselines, so it is the wrong headline for a redactor. The leak-rate scorecard above answers recall ("did we miss real PII?"); this one answers the inverse cost โ€” over-redaction. Same structure as the leak scorecard: roll-up rows only (per domain ยท per language ยท overall), `anonde-ner` anchored first. ๐Ÿฅ‡ marks the highest-precision engine in the row. Each cell pools tp/(tp+fp) across the group (micro-average, doc-weighted) and annotates the pooled raw FP count โ€” precision can look fine while absolute false-positive volume is high. -| Slice | Scope | `anonde-ner` โฌ…๏ธŽ anonde (default NER) | `anonde-patterns` | `presidio` | `presidio-transformer` | `gliner-py` | `openai-pf` | -|---|---|---:|---:|---:|---:|---:|---:| -| **ฮฃ ALL** | **all** | **67.1% (31924 fp)** | **76.9% (12287 fp)** | **58.2% (26886 fp)** | **74.2% (2969 fp)** | **79.2% (12753 fp)** ๐Ÿฅ‡ | **75.2% (4690 fp)** | -| | | | | | | | | -| _ฮฃ Clinical / medical de-identification_ | _all langs_ | 60.9% (8527 fp) | 72.0% (3197 fp) | 53.2% (8508 fp) | 85.8% (303 fp) | **88.5% (1491 fp)** ๐Ÿฅ‡ | 88.4% (736 fp) | -| _ฮฃ Legal / administrative_ | _all langs_ | 43.2% (4153 fp) | **81.4% (587 fp)** ๐Ÿฅ‡ | 47.4% (2242 fp) | 25.7% (465 fp) | 45.5% (3315 fp) | 75.9% (228 fp) | -| _ฮฃ Retail finance_ | _all langs_ | 83.2% (2275 fp) | 84.9% (1649 fp) | 79.4% (2391 fp) | **93.1% (123 fp)** ๐Ÿฅ‡ | 92.0% (957 fp) | 83.5% (877 fp) | -| _ฮฃ Enterprise logs_ | _all langs_ | 73.2% (1034 fp) | **83.7% (502 fp)** ๐Ÿฅ‡ | 57.3% (1554 fp) | 59.5% (1418 fp) | 75.9% (243 fp) | 30.2% (2497 fp) | -| _ฮฃ General structured PII_ | _all langs_ | 67.8% (13376 fp) | 71.0% (5199 fp) | 54.5% (8861 fp) | 81.8% (527 fp) | 78.3% (5131 fp) | **93.1% (113 fp)** ๐Ÿฅ‡ | -| _ฮฃ Academic NER (newswire / social)_ | _all langs_ | 38.6% (1479 fp) | 41.3% (334 fp) | 71.5% (378 fp) | **77.6% (133 fp)** ๐Ÿฅ‡ | 57.9% (829 fp) | 70.0% (36 fp) | -| _ฮฃ Adversarial / out-of-distribution_ | _all langs_ | 83.2% (1080 fp) | **86.5% (819 fp)** ๐Ÿฅ‡ | 49.5% (2952 fp) | โ€“ | 80.0% (787 fp) | 80.4% (203 fp) | -| | | | | | | | | -| _ฮฃ all domains_ | _English_ | 56.5% (9568 fp) | 57.2% (6800 fp) | 65.2% (4281 fp) | 74.2% (2969 fp) | **79.4% (2240 fp)** ๐Ÿฅ‡ | 53.4% (2694 fp) | -| _ฮฃ all domains_ | _German_ | 68.9% (10123 fp) | 79.9% (4623 fp) | 58.8% (9013 fp) | โ€“ | 79.3% (4336 fp) | **85.7% (932 fp)** ๐Ÿฅ‡ | -| _ฮฃ all domains_ | _Spanish_ | 68.0% (5187 fp) | **97.1% (127 fp)** ๐Ÿฅ‡ | 51.6% (5756 fp) | โ€“ | 80.9% (2131 fp) | 76.5% (504 fp) | -| _ฮฃ all domains_ | _French_ | 75.9% (3080 fp) | **95.8% (191 fp)** ๐Ÿฅ‡ | 56.5% (4116 fp) | โ€“ | 81.0% (1782 fp) | 87.5% (270 fp) | -| _ฮฃ all domains_ | _Italian_ | 70.3% (3966 fp) | **89.7% (546 fp)** ๐Ÿฅ‡ | 57.7% (3720 fp) | โ€“ | 75.1% (2264 fp) | 87.5% (290 fp) | +| Slice | Scope | `anonde-ner` โฌ…๏ธŽ anonde (default NER) | `anonde-patterns` | `presidio` | `gliner-py` | `openai-pf` | +|---|---|---:|---:|---:|---:|---:| +| **ฮฃ ALL** | **all** | **67.1% (31928 fp)** | **76.9% (12290 fp)** | **58.2% (26878 fp)** | **79.2% (12759 fp)** ๐Ÿฅ‡ | **75.2% (4688 fp)** | +| | | | | | | | +| _ฮฃ Clinical / medical de-identification_ | _all langs_ | 60.9% (8527 fp) | 72.0% (3197 fp) | 53.2% (8508 fp) | **88.5% (1491 fp)** ๐Ÿฅ‡ | 88.3% (738 fp) | +| _ฮฃ Legal / administrative_ | _all langs_ | 43.2% (4153 fp) | **81.4% (587 fp)** ๐Ÿฅ‡ | 47.4% (2242 fp) | 45.5% (3315 fp) | 75.9% (228 fp) | +| _ฮฃ Retail finance_ | _all langs_ | 83.2% (2276 fp) | 84.9% (1650 fp) | 79.4% (2391 fp) | **92.0% (957 fp)** ๐Ÿฅ‡ | 83.5% (877 fp) | +| _ฮฃ Enterprise logs_ | _all langs_ | 73.2% (1034 fp) | **83.7% (502 fp)** ๐Ÿฅ‡ | 57.3% (1554 fp) | 75.9% (243 fp) | 30.1% (2492 fp) | +| _ฮฃ General structured PII_ | _all langs_ | 67.8% (13376 fp) | 71.0% (5199 fp) | 54.5% (8861 fp) | 78.3% (5131 fp) | **93.1% (113 fp)** ๐Ÿฅ‡ | +| _ฮฃ Academic NER (newswire / social)_ | _all langs_ | 38.6% (1479 fp) | 41.3% (334 fp) | **71.5% (378 fp)** ๐Ÿฅ‡ | 57.9% (829 fp) | 69.4% (37 fp) | +| _ฮฃ Adversarial / out-of-distribution_ | _all langs_ | 83.2% (1083 fp) | **86.5% (821 fp)** ๐Ÿฅ‡ | 49.7% (2944 fp) | 79.9% (793 fp) | 80.4% (203 fp) | +| | | | | | | | +| _ฮฃ all domains_ | _English_ | 56.5% (9568 fp) | 57.2% (6800 fp) | 65.2% (4281 fp) | **79.4% (2240 fp)** ๐Ÿฅ‡ | 53.4% (2690 fp) | +| _ฮฃ all domains_ | _German_ | 68.9% (10127 fp) | 79.9% (4626 fp) | 58.9% (9005 fp) | 79.2% (4342 fp) | **85.7% (935 fp)** ๐Ÿฅ‡ | +| _ฮฃ all domains_ | _Spanish_ | 68.0% (5187 fp) | **97.1% (127 fp)** ๐Ÿฅ‡ | 51.6% (5756 fp) | 80.9% (2131 fp) | 76.6% (503 fp) | +| _ฮฃ all domains_ | _French_ | 75.9% (3080 fp) | **95.8% (191 fp)** ๐Ÿฅ‡ | 56.5% (4116 fp) | 81.0% (1782 fp) | 87.5% (270 fp) | +| _ฮฃ all domains_ | _Italian_ | 70.3% (3966 fp) | **89.7% (546 fp)** ๐Ÿฅ‡ | 57.7% (3720 fp) | 75.1% (2264 fp) | 87.5% (290 fp) | > **Reading this table** โ€” a cell of `92.0% (40 fp)` means 92% of the spans that engine redacted overlapped real PII; the remaining 8% (40 absolute spans) were over-redaction. Recall (leak rate) is in the scorecard above; this is the other half of the trade-off. -> **Why some predictions are not counted** โ€” a `(corpus, entity-type)` cell where the gold annotates **zero** spans of that type (`tp + fn == 0`) is *unscoreable for precision*: with no gold of that type present, every prediction there is mechanically a false positive against absent gold โ€” a **schema gap** (e.g. a corpus that annotates PERSON but not LOCATION), not real over-redaction. Such cells are **excluded** from the precision pool above, and an empty-gold corpus (every type zero-gold) drops out entirely. This is a scorecard-aggregation choice only โ€” the raw per-type counts stay intact in `results_matrix.csv`, and **leak-rate / recall are untouched** (they score against the full gold). Excluded here: **125 (corpus, type) cells** across **1 empty-gold corpora**. For full transparency, the raw ฮฃ ALL precision *including* every zero-gold cell: `anonde-ner` 60.2% (42906 fp) ยท `anonde-patterns` 73.9% (14435 fp) ยท `presidio` 47.9% (40742 fp) ยท `presidio-transformer` 59.7% (5768 fp) ยท `gliner-py` 67.5% (23463 fp) ยท `openai-pf` 69.6% (6213 fp). +> **Why some predictions are not counted** โ€” a `(corpus, entity-type)` cell where the gold annotates **zero** spans of that type (`tp + fn == 0`) is *unscoreable for precision*: with no gold of that type present, every prediction there is mechanically a false positive against absent gold โ€” a **schema gap** (e.g. a corpus that annotates PERSON but not LOCATION), not real over-redaction. Such cells are **excluded** from the precision pool above, and an empty-gold corpus (every type zero-gold) drops out entirely. This is a scorecard-aggregation choice only โ€” the raw per-type counts stay intact in `results_matrix.csv`, and **leak-rate / recall are untouched** (they score against the full gold). Excluded here: **125 (corpus, type) cells** across **1 empty-gold corpora**. For full transparency, the raw ฮฃ ALL precision *including* every zero-gold cell: `anonde-ner` 60.2% (42910 fp) ยท `anonde-patterns` 73.9% (14438 fp) ยท `presidio` 47.9% (40734 fp) ยท `gliner-py` 67.5% (23469 fp) ยท `openai-pf` 69.6% (6218 fp).
Engine profiles ยท what each column means @@ -123,63 +121,63 @@ Everything below is reference detail behind the scorecard. The per-cell grid fir Detail behind the scorecard roll-ups: one row per populated `(domain, language)` cell. Same columns, same anchor, same verdict glyph โ€” read this to see *which* baseline wins where. Pooled leak rate across the cell's corpora. -| Domain | Language | `anonde-ner` โฌ…๏ธŽ anonde (default NER) | `anonde-patterns` | `presidio` | `presidio-transformer` | `gliner-py` | `openai-pf` | Verdict | -|---|---|---:|---:|---:|---:|---:|---:|:--:| -| **Clinical / medical de-identification** | English | **1.6%** ๐Ÿฅ‡ | 8.0% | 20.3% | 15.2% | 23.3% | 24.6% | โœ… | -| **Clinical / medical de-identification** | German | **4.6%** ๐Ÿฅ‡ | 9.0% | 30.9% | โ€“ | 34.2% | 29.9% | โœ… | -| **Clinical / medical de-identification** | Spanish | **17.8%** ๐Ÿฅ‡ | 79.6% | 38.4% | โ€“ | 23.7% | 31.6% | โœ… | -| **Clinical / medical de-identification** | French | **11.5%** ๐Ÿฅ‡ | 61.4% | 28.2% | โ€“ | 25.3% | 21.8% | โœ… | -| **Clinical / medical de-identification** | Italian | **16.2%** ๐Ÿฅ‡ | 59.7% | 28.5% | โ€“ | 35.2% | 25.1% | โœ… | -| **Legal / administrative** | English | 8.4% | 48.9% | **6.2%** ๐Ÿฅ‡ | 13.3% | 10.7% | 76.9% | โŒ | -| **Legal / administrative** | German | **1.1%** ๐Ÿฅ‡ | 11.4% | 26.7% | โ€“ | 23.8% | 32.9% | โœ… | -| **Legal / administrative** | Spanish | **6.5%** ๐Ÿฅ‡ | 100.0% | 59.1% | โ€“ | 18.3% | 93.8% | โœ… | -| **Legal / administrative** | French | **11.4%** ๐Ÿฅ‡ | 75.2% | 44.9% | โ€“ | 16.3% | 95.5% | โœ… | -| **Legal / administrative** | Italian | **5.5%** ๐Ÿฅ‡ | 40.4% | 67.1% | โ€“ | 6.2% | 50.0% | โœ… | -| **Retail finance** | English | **2.8%** ๐Ÿฅ‡ | 21.1% | 10.3% | 12.0% | 20.6% | 18.5% | โœ… | -| **Retail finance** | German | **2.6%** ๐Ÿฅ‡ | 3.4% | 24.1% | โ€“ | 22.6% | 21.3% | โœ… | -| **Retail finance** | Spanish | **16.4%** ๐Ÿฅ‡ | 46.4% | 18.5% | โ€“ | 25.4% | 17.4% | โœ… | -| **Retail finance** | French | **15.9%** ๐Ÿฅ‡ | 44.6% | 19.4% | โ€“ | 23.9% | 19.3% | โœ… | -| **Retail finance** | Italian | **11.7%** ๐Ÿฅ‡ | 39.5% | 29.6% | โ€“ | 26.4% | 14.1% | โœ… | -| **Enterprise logs** | English | **13.2%** ๐Ÿฅ‡ | 28.9% | 31.5% | 37.3% | 73.2% | 15.3% | โœ… | -| **General structured PII** | English | **13.0%** ๐Ÿฅ‡ | 47.8% | 56.0% | 57.4% | 35.0% | 15.5% | โœ… | -| **General structured PII** | German | **10.7%** ๐Ÿฅ‡ | 59.8% | 58.4% | โ€“ | 33.0% | 23.6% | โœ… | -| **General structured PII** | Spanish | **14.2%** ๐Ÿฅ‡ | 68.9% | 61.0% | โ€“ | 34.6% | 17.5% | โœ… | -| **General structured PII** | French | **14.2%** ๐Ÿฅ‡ | 68.9% | 54.0% | โ€“ | 34.9% | 16.3% | โœ… | -| **General structured PII** | Italian | **15.0%** ๐Ÿฅ‡ | 63.9% | 59.6% | โ€“ | 36.0% | 21.2% | โœ… | -| **Academic NER (newswire / social)** | English | **6.6%** ๐Ÿฅ‡ | 92.0% | 17.4% | 16.5% | 11.8% | 71.9% | โœ… | -| **Academic NER (newswire / social)** | German | **5.8%** ๐Ÿฅ‡ | 47.7% | 19.1% | โ€“ | 15.5% | 73.4% | โœ… | -| **Adversarial / out-of-distribution** | German | **8.1%** ๐Ÿฅ‡ | 12.7% | 37.6% | โ€“ | 43.4% | 32.0% | โœ… | +| Domain | Language | `anonde-ner` โฌ…๏ธŽ anonde (default NER) | `anonde-patterns` | `presidio` | `gliner-py` | `openai-pf` | Verdict | +|---|---|---:|---:|---:|---:|---:|:--:| +| **Clinical / medical de-identification** | English | **1.6%** ๐Ÿฅ‡ | 8.0% | 20.3% | 23.3% | 24.6% | โœ… | +| **Clinical / medical de-identification** | German | **4.6%** ๐Ÿฅ‡ | 9.0% | 30.9% | 34.2% | 30.0% | โœ… | +| **Clinical / medical de-identification** | Spanish | **17.8%** ๐Ÿฅ‡ | 79.6% | 38.4% | 23.7% | 31.6% | โœ… | +| **Clinical / medical de-identification** | French | **11.5%** ๐Ÿฅ‡ | 61.4% | 28.2% | 25.3% | 21.8% | โœ… | +| **Clinical / medical de-identification** | Italian | **16.2%** ๐Ÿฅ‡ | 59.7% | 28.5% | 35.2% | 25.1% | โœ… | +| **Legal / administrative** | English | 8.4% | 48.9% | **6.2%** ๐Ÿฅ‡ | 10.7% | 76.9% | โŒ | +| **Legal / administrative** | German | **1.1%** ๐Ÿฅ‡ | 11.4% | 26.7% | 23.8% | 32.9% | โœ… | +| **Legal / administrative** | Spanish | **6.5%** ๐Ÿฅ‡ | 100.0% | 59.1% | 18.3% | 93.8% | โœ… | +| **Legal / administrative** | French | **11.4%** ๐Ÿฅ‡ | 75.2% | 44.9% | 16.3% | 95.5% | โœ… | +| **Legal / administrative** | Italian | **5.5%** ๐Ÿฅ‡ | 40.4% | 67.1% | 6.2% | 50.0% | โœ… | +| **Retail finance** | English | **2.8%** ๐Ÿฅ‡ | 21.1% | 10.3% | 20.6% | 18.5% | โœ… | +| **Retail finance** | German | **2.6%** ๐Ÿฅ‡ | 3.4% | 24.1% | 22.6% | 21.1% | โœ… | +| **Retail finance** | Spanish | **16.4%** ๐Ÿฅ‡ | 46.4% | 18.5% | 25.4% | 17.6% | โœ… | +| **Retail finance** | French | **15.9%** ๐Ÿฅ‡ | 44.6% | 19.4% | 23.9% | 19.3% | โœ… | +| **Retail finance** | Italian | **11.7%** ๐Ÿฅ‡ | 39.5% | 29.6% | 26.4% | 14.1% | โœ… | +| **Enterprise logs** | English | **13.2%** ๐Ÿฅ‡ | 28.9% | 31.5% | 73.2% | 15.2% | โœ… | +| **General structured PII** | English | **13.0%** ๐Ÿฅ‡ | 47.8% | 56.0% | 35.0% | 15.5% | โœ… | +| **General structured PII** | German | **10.7%** ๐Ÿฅ‡ | 59.8% | 58.4% | 33.0% | 23.6% | โœ… | +| **General structured PII** | Spanish | **14.2%** ๐Ÿฅ‡ | 68.9% | 61.0% | 34.6% | 17.5% | โœ… | +| **General structured PII** | French | **14.2%** ๐Ÿฅ‡ | 68.9% | 54.0% | 34.9% | 16.3% | โœ… | +| **General structured PII** | Italian | **15.0%** ๐Ÿฅ‡ | 63.9% | 59.6% | 36.0% | 21.2% | โœ… | +| **Academic NER (newswire / social)** | English | **6.6%** ๐Ÿฅ‡ | 92.0% | 17.4% | 11.8% | 71.9% | โœ… | +| **Academic NER (newswire / social)** | German | **5.8%** ๐Ÿฅ‡ | 47.7% | 19.1% | 15.5% | 73.4% | โœ… | +| **Adversarial / out-of-distribution** | German | **8.0%** ๐Ÿฅ‡ | 12.6% | 37.5% | 43.3% | 32.0% | โœ… | ## Per-cell precision ยท domain ร— language Detail behind the precision scorecard: one row per populated `(domain, language)` cell, partial precision pooled across the cell's corpora (raw FP count annotated). Higher is better; ๐Ÿฅ‡ marks the highest-precision engine in the row. Strict byte-exact precision per entity type stays in `results_matrix.csv`. -| Domain | Language | `anonde-ner` โฌ…๏ธŽ anonde (default NER) | `anonde-patterns` | `presidio` | `presidio-transformer` | `gliner-py` | `openai-pf` | -|---|---|---:|---:|---:|---:|---:|---:| -| **Clinical / medical de-identification** | English | 51.9% (2035 fp) | 51.2% (1929 fp) | 78.3% (458 fp) | 85.8% (303 fp) | 93.9% (126 fp) | **99.1% (9 fp)** ๐Ÿฅ‡ | -| **Clinical / medical de-identification** | German | 53.2% (2905 fp) | 74.2% (1158 fp) | 44.6% (2796 fp) | โ€“ | **87.7% (334 fp)** ๐Ÿฅ‡ | 87.5% (273 fp) | -| **Clinical / medical de-identification** | Spanish | 58.2% (2759 fp) | **98.5% (16 fp)** ๐Ÿฅ‡ | 44.0% (3307 fp) | โ€“ | 82.1% (806 fp) | 69.8% (313 fp) | -| **Clinical / medical de-identification** | French | 82.8% (421 fp) | **97.4% (24 fp)** ๐Ÿฅ‡ | 59.4% (966 fp) | โ€“ | 97.0% (58 fp) | 92.1% (83 fp) | -| **Clinical / medical de-identification** | Italian | 82.4% (407 fp) | 92.9% (70 fp) | 64.2% (981 fp) | โ€“ | 90.5% (167 fp) | **94.5% (58 fp)** ๐Ÿฅ‡ | -| **Legal / administrative** | English | 17.0% (767 fp) | 58.6% (67 fp) | 18.9% (693 fp) | 25.7% (465 fp) | 21.7% (637 fp) | **100.0% (0 fp)** ๐Ÿฅ‡ | -| **Legal / administrative** | German | 65.9% (1344 fp) | **81.7% (518 fp)** ๐Ÿฅ‡ | 67.0% (842 fp) | โ€“ | 65.8% (1135 fp) | 78.9% (189 fp) | -| **Legal / administrative** | Spanish | 18.3% (259 fp) | โ€“ | 6.5% (145 fp) | โ€“ | **23.6% (194 fp)** ๐Ÿฅ‡ | 11.1% (16 fp) | -| **Legal / administrative** | French | 22.2% (809 fp) | **98.8% (1 fp)** ๐Ÿฅ‡ | 35.6% (224 fp) | โ€“ | 28.0% (582 fp) | 0.0% (8 fp) | -| **Legal / administrative** | Italian | 10.0% (974 fp) | **98.8% (1 fp)** ๐Ÿฅ‡ | 5.1% (338 fp) | โ€“ | 13.2% (767 fp) | 28.6% (15 fp) | -| **Retail finance** | English | 69.3% (807 fp) | 63.0% (833 fp) | 80.7% (398 fp) | **93.1% (123 fp)** ๐Ÿฅ‡ | 91.7% (161 fp) | 82.1% (137 fp) | -| **Retail finance** | German | 80.2% (1121 fp) | 85.1% (785 fp) | 78.9% (891 fp) | โ€“ | **90.0% (460 fp)** ๐Ÿฅ‡ | 88.6% (238 fp) | -| **Retail finance** | Spanish | 93.9% (102 fp) | **99.2% (8 fp)** ๐Ÿฅ‡ | 78.8% (397 fp) | โ€“ | 94.5% (96 fp) | 80.1% (159 fp) | -| **Retail finance** | French | 94.8% (92 fp) | **100.0% (0 fp)** ๐Ÿฅ‡ | 82.8% (317 fp) | โ€“ | 93.5% (122 fp) | 80.3% (161 fp) | -| **Retail finance** | Italian | 91.8% (153 fp) | **98.1% (23 fp)** ๐Ÿฅ‡ | 76.0% (388 fp) | โ€“ | 93.2% (118 fp) | 78.5% (182 fp) | -| **Enterprise logs** | English | 73.2% (1034 fp) | **83.7% (502 fp)** ๐Ÿฅ‡ | 57.3% (1554 fp) | 59.5% (1418 fp) | 75.9% (243 fp) | 30.2% (2497 fp) | -| **General structured PII** | English | 53.8% (4273 fp) | 46.6% (3359 fp) | 67.8% (970 fp) | 81.8% (527 fp) | 83.4% (696 fp) | **92.3% (30 fp)** ๐Ÿฅ‡ | -| **General structured PII** | German | 68.5% (2846 fp) | 71.6% (1119 fp) | 61.4% (1362 fp) | โ€“ | 77.8% (1168 fp) | **94.6% (14 fp)** ๐Ÿฅ‡ | -| **General structured PII** | Spanish | 72.9% (2067 fp) | **95.5% (103 fp)** ๐Ÿฅ‡ | 51.8% (1907 fp) | โ€“ | 77.8% (1035 fp) | 94.5% (16 fp) | -| **General structured PII** | French | 76.8% (1758 fp) | 93.1% (166 fp) | 46.7% (2609 fp) | โ€“ | 78.7% (1020 fp) | **93.6% (18 fp)** ๐Ÿฅ‡ | -| **General structured PII** | Italian | 69.9% (2432 fp) | 84.9% (452 fp) | 50.6% (2013 fp) | โ€“ | 74.2% (1212 fp) | **91.3% (35 fp)** ๐Ÿฅ‡ | -| **Academic NER (newswire / social)** | English | 40.5% (652 fp) | 19.1% (110 fp) | 67.2% (208 fp) | **77.6% (133 fp)** ๐Ÿฅ‡ | 56.6% (377 fp) | 63.8% (21 fp) | -| **Academic NER (newswire / social)** | German | 37.0% (827 fp) | 48.3% (224 fp) | 75.4% (170 fp) | โ€“ | 58.9% (452 fp) | **75.8% (15 fp)** ๐Ÿฅ‡ | -| **Adversarial / out-of-distribution** | German | 83.2% (1080 fp) | **86.5% (819 fp)** ๐Ÿฅ‡ | 49.5% (2952 fp) | โ€“ | 80.0% (787 fp) | 80.4% (203 fp) | +| Domain | Language | `anonde-ner` โฌ…๏ธŽ anonde (default NER) | `anonde-patterns` | `presidio` | `gliner-py` | `openai-pf` | +|---|---|---:|---:|---:|---:|---:| +| **Clinical / medical de-identification** | English | 51.9% (2035 fp) | 51.2% (1929 fp) | 78.3% (458 fp) | 93.9% (126 fp) | **99.1% (9 fp)** ๐Ÿฅ‡ | +| **Clinical / medical de-identification** | German | 53.2% (2905 fp) | 74.2% (1158 fp) | 44.6% (2796 fp) | **87.7% (334 fp)** ๐Ÿฅ‡ | 87.4% (275 fp) | +| **Clinical / medical de-identification** | Spanish | 58.2% (2759 fp) | **98.5% (16 fp)** ๐Ÿฅ‡ | 44.0% (3307 fp) | 82.1% (806 fp) | 69.8% (313 fp) | +| **Clinical / medical de-identification** | French | 82.8% (421 fp) | **97.4% (24 fp)** ๐Ÿฅ‡ | 59.4% (966 fp) | 97.0% (58 fp) | 92.1% (83 fp) | +| **Clinical / medical de-identification** | Italian | 82.4% (407 fp) | 92.9% (70 fp) | 64.2% (981 fp) | 90.5% (167 fp) | **94.5% (58 fp)** ๐Ÿฅ‡ | +| **Legal / administrative** | English | 17.0% (767 fp) | 58.6% (67 fp) | 18.9% (693 fp) | 21.7% (637 fp) | **100.0% (0 fp)** ๐Ÿฅ‡ | +| **Legal / administrative** | German | 65.9% (1344 fp) | **81.7% (518 fp)** ๐Ÿฅ‡ | 67.0% (842 fp) | 65.8% (1135 fp) | 78.9% (189 fp) | +| **Legal / administrative** | Spanish | 18.3% (259 fp) | โ€“ | 6.5% (145 fp) | **23.6% (194 fp)** ๐Ÿฅ‡ | 11.1% (16 fp) | +| **Legal / administrative** | French | 22.2% (809 fp) | **98.8% (1 fp)** ๐Ÿฅ‡ | 35.6% (224 fp) | 28.0% (582 fp) | 0.0% (8 fp) | +| **Legal / administrative** | Italian | 10.0% (974 fp) | **98.8% (1 fp)** ๐Ÿฅ‡ | 5.1% (338 fp) | 13.2% (767 fp) | 28.6% (15 fp) | +| **Retail finance** | English | 69.3% (807 fp) | 63.0% (833 fp) | 80.7% (398 fp) | **91.7% (161 fp)** ๐Ÿฅ‡ | 82.1% (137 fp) | +| **Retail finance** | German | 80.1% (1122 fp) | 85.1% (786 fp) | 78.9% (891 fp) | **90.0% (460 fp)** ๐Ÿฅ‡ | 88.6% (239 fp) | +| **Retail finance** | Spanish | 93.9% (102 fp) | **99.2% (8 fp)** ๐Ÿฅ‡ | 78.8% (397 fp) | 94.5% (96 fp) | 80.2% (158 fp) | +| **Retail finance** | French | 94.8% (92 fp) | **100.0% (0 fp)** ๐Ÿฅ‡ | 82.8% (317 fp) | 93.5% (122 fp) | 80.3% (161 fp) | +| **Retail finance** | Italian | 91.8% (153 fp) | **98.1% (23 fp)** ๐Ÿฅ‡ | 76.0% (388 fp) | 93.2% (118 fp) | 78.5% (182 fp) | +| **Enterprise logs** | English | 73.2% (1034 fp) | **83.7% (502 fp)** ๐Ÿฅ‡ | 57.3% (1554 fp) | 75.9% (243 fp) | 30.1% (2492 fp) | +| **General structured PII** | English | 53.8% (4273 fp) | 46.6% (3359 fp) | 67.8% (970 fp) | 83.4% (696 fp) | **92.3% (30 fp)** ๐Ÿฅ‡ | +| **General structured PII** | German | 68.5% (2846 fp) | 71.6% (1119 fp) | 61.4% (1362 fp) | 77.8% (1168 fp) | **94.6% (14 fp)** ๐Ÿฅ‡ | +| **General structured PII** | Spanish | 72.9% (2067 fp) | **95.5% (103 fp)** ๐Ÿฅ‡ | 51.8% (1907 fp) | 77.8% (1035 fp) | 94.5% (16 fp) | +| **General structured PII** | French | 76.8% (1758 fp) | 93.1% (166 fp) | 46.7% (2609 fp) | 78.7% (1020 fp) | **93.6% (18 fp)** ๐Ÿฅ‡ | +| **General structured PII** | Italian | 69.9% (2432 fp) | 84.9% (452 fp) | 50.6% (2013 fp) | 74.2% (1212 fp) | **91.3% (35 fp)** ๐Ÿฅ‡ | +| **Academic NER (newswire / social)** | English | 40.5% (652 fp) | 19.1% (110 fp) | **67.2% (208 fp)** ๐Ÿฅ‡ | 56.6% (377 fp) | 62.7% (22 fp) | +| **Academic NER (newswire / social)** | German | 37.0% (827 fp) | 48.3% (224 fp) | 75.4% (170 fp) | 58.9% (452 fp) | **75.8% (15 fp)** ๐Ÿฅ‡ | +| **Adversarial / out-of-distribution** | German | 83.2% (1083 fp) | **86.5% (821 fp)** ๐Ÿฅ‡ | 49.7% (2944 fp) | 79.9% (793 fp) | 80.4% (203 fp) | ## Clinical / medical de-identification ยท English @@ -189,9 +187,9 @@ Corpora in this group: `synth_clinical_en`. A gold PHI span is *leaked* when **no** predicted span overlaps it โ€” 'did we miss a name?' -| Corpus | `anonde-patterns` | `anonde-ner` | `presidio` | `presidio-transformer` | `gliner-py` | `openai-pf` | -|---|---:|---:|---:|---:|---:|---:| -| `synth_clinical_en` | 8.0% | **1.6%** ๐Ÿฅ‡ | 20.3% | 15.2% | 23.3% | 24.6% | +| Corpus | `anonde-patterns` | `anonde-ner` | `presidio` | `gliner-py` | `openai-pf` | +|---|---:|---:|---:|---:|---:| +| `synth_clinical_en` | 8.0% | **1.6%** ๐Ÿฅ‡ | 20.3% | 23.3% | 24.6% | > **Partial coverage** โ€” some engines were benchmarked on a fixed subsample, not every gold doc: > @@ -201,9 +199,9 @@ A gold PHI span is *leaked* when **no** predicted span overlaps it โ€” 'did we m Each leaked span weighted by compliance tier โ€” direct identifiers (PERSON, EMAIL, PHONE, ADDRESS, DOB) = 5, high-stakes IDs (SSN/MRN/IBAN) = 10, quasi-identifiers (LOCATION, ORG, PROFESSION) = 1. Defaults in `label_map.yaml::severity`. Shown only because at least one cell here moves >3pp from raw leak; otherwise the two tables tracked within noise. -| Corpus | `anonde-patterns` | `anonde-ner` | `presidio` | `presidio-transformer` | `gliner-py` | `openai-pf` | -|---|---:|---:|---:|---:|---:|---:| -| `synth_clinical_en` | 5.3% | **1.7%** ๐Ÿฅ‡ | 26.7% | 21.3% | 26.4% | 17.4% | +| Corpus | `anonde-patterns` | `anonde-ner` | `presidio` | `gliner-py` | `openai-pf` | +|---|---:|---:|---:|---:|---:| +| `synth_clinical_en` | 5.3% | **1.7%** ๐Ÿฅ‡ | 26.7% | 26.4% | 17.4% | ## Clinical / medical de-identification ยท German @@ -213,10 +211,10 @@ Corpora in this group: `openmed`, `pmc_de`, `synth_clinical`, `wiki_de`. A gold PHI span is *leaked* when **no** predicted span overlaps it โ€” 'did we miss a name?' -| Corpus | `anonde-patterns` | `anonde-ner` | `presidio` | `presidio-transformer` | `gliner-py` | `openai-pf` | -|---|---:|---:|---:|---:|---:|---:| -| `openmed` | 21.6% | **11.7%** ๐Ÿฅ‡ | 33.9% | โ€“ | 49.7% | 35.4% | -| `synth_clinical` | 1.8% | **0.5%** ๐Ÿฅ‡ | 29.1% | โ€“ | 25.4% | 23.4% | +| Corpus | `anonde-patterns` | `anonde-ner` | `presidio` | `gliner-py` | `openai-pf` | +|---|---:|---:|---:|---:|---:| +| `openmed` | 21.6% | **11.7%** ๐Ÿฅ‡ | 33.9% | 49.7% | 35.4% | +| `synth_clinical` | 1.8% | **0.5%** ๐Ÿฅ‡ | 29.1% | 25.4% | 23.5% | > **Partial coverage** โ€” some engines were benchmarked on a fixed subsample, not every gold doc: > @@ -227,10 +225,10 @@ A gold PHI span is *leaked* when **no** predicted span overlaps it โ€” 'did we m Each leaked span weighted by compliance tier โ€” direct identifiers (PERSON, EMAIL, PHONE, ADDRESS, DOB) = 5, high-stakes IDs (SSN/MRN/IBAN) = 10, quasi-identifiers (LOCATION, ORG, PROFESSION) = 1. Defaults in `label_map.yaml::severity`. Shown only because at least one cell here moves >3pp from raw leak; otherwise the two tables tracked within noise. -| Corpus | `anonde-patterns` | `anonde-ner` | `presidio` | `presidio-transformer` | `gliner-py` | `openai-pf` | -|---|---:|---:|---:|---:|---:|---:| -| `openmed` | 21.1% | **11.6%** ๐Ÿฅ‡ | 35.5% | โ€“ | 50.5% | 33.8% | -| `synth_clinical` | 0.4% | **0.1%** ๐Ÿฅ‡ | 33.3% | โ€“ | 27.5% | 17.4% | +| Corpus | `anonde-patterns` | `anonde-ner` | `presidio` | `gliner-py` | `openai-pf` | +|---|---:|---:|---:|---:|---:| +| `openmed` | 21.1% | **11.6%** ๐Ÿฅ‡ | 35.5% | 50.5% | 33.8% | +| `synth_clinical` | 0.4% | **0.1%** ๐Ÿฅ‡ | 33.3% | 27.5% | 17.6% | ## Clinical / medical de-identification ยท Spanish @@ -240,9 +238,9 @@ Corpora in this group: `pharmaconer_es`, `meddocan_es`. A gold PHI span is *leaked* when **no** predicted span overlaps it โ€” 'did we miss a name?' -| Corpus | `anonde-patterns` | `anonde-ner` | `presidio` | `presidio-transformer` | `gliner-py` | `openai-pf` | -|---|---:|---:|---:|---:|---:|---:| -| `meddocan_es` | 79.6% | **17.8%** ๐Ÿฅ‡ | 38.4% | โ€“ | 23.7% | 31.6% | +| Corpus | `anonde-patterns` | `anonde-ner` | `presidio` | `gliner-py` | `openai-pf` | +|---|---:|---:|---:|---:|---:| +| `meddocan_es` | 79.6% | **17.8%** ๐Ÿฅ‡ | 38.4% | 23.7% | 31.6% | > **Partial coverage** โ€” some engines were benchmarked on a fixed subsample, not every gold doc: > @@ -253,9 +251,9 @@ A gold PHI span is *leaked* when **no** predicted span overlaps it โ€” 'did we m Each leaked span weighted by compliance tier โ€” direct identifiers (PERSON, EMAIL, PHONE, ADDRESS, DOB) = 5, high-stakes IDs (SSN/MRN/IBAN) = 10, quasi-identifiers (LOCATION, ORG, PROFESSION) = 1. Defaults in `label_map.yaml::severity`. Shown only because at least one cell here moves >3pp from raw leak; otherwise the two tables tracked within noise. -| Corpus | `anonde-patterns` | `anonde-ner` | `presidio` | `presidio-transformer` | `gliner-py` | `openai-pf` | -|---|---:|---:|---:|---:|---:|---:| -| `meddocan_es` | 75.4% | 21.2% | 46.4% | โ€“ | 27.2% | **14.1%** ๐Ÿฅ‡ | +| Corpus | `anonde-patterns` | `anonde-ner` | `presidio` | `gliner-py` | `openai-pf` | +|---|---:|---:|---:|---:|---:| +| `meddocan_es` | 75.4% | 21.2% | 46.4% | 27.2% | **14.1%** ๐Ÿฅ‡ | ## Clinical / medical de-identification ยท French @@ -265,9 +263,9 @@ Corpora in this group: `synth_clinical_fr`. A gold PHI span is *leaked* when **no** predicted span overlaps it โ€” 'did we miss a name?' -| Corpus | `anonde-patterns` | `anonde-ner` | `presidio` | `presidio-transformer` | `gliner-py` | `openai-pf` | -|---|---:|---:|---:|---:|---:|---:| -| `synth_clinical_fr` | 61.4% | **11.5%** ๐Ÿฅ‡ | 28.2% | โ€“ | 25.3% | 21.8% | +| Corpus | `anonde-patterns` | `anonde-ner` | `presidio` | `gliner-py` | `openai-pf` | +|---|---:|---:|---:|---:|---:| +| `synth_clinical_fr` | 61.4% | **11.5%** ๐Ÿฅ‡ | 28.2% | 25.3% | 21.8% | > **Partial coverage** โ€” some engines were benchmarked on a fixed subsample, not every gold doc: > @@ -277,9 +275,9 @@ A gold PHI span is *leaked* when **no** predicted span overlaps it โ€” 'did we m Each leaked span weighted by compliance tier โ€” direct identifiers (PERSON, EMAIL, PHONE, ADDRESS, DOB) = 5, high-stakes IDs (SSN/MRN/IBAN) = 10, quasi-identifiers (LOCATION, ORG, PROFESSION) = 1. Defaults in `label_map.yaml::severity`. Shown only because at least one cell here moves >3pp from raw leak; otherwise the two tables tracked within noise. -| Corpus | `anonde-patterns` | `anonde-ner` | `presidio` | `presidio-transformer` | `gliner-py` | `openai-pf` | -|---|---:|---:|---:|---:|---:|---:| -| `synth_clinical_fr` | 56.8% | **11.6%** ๐Ÿฅ‡ | 30.9% | โ€“ | 28.0% | 15.3% | +| Corpus | `anonde-patterns` | `anonde-ner` | `presidio` | `gliner-py` | `openai-pf` | +|---|---:|---:|---:|---:|---:| +| `synth_clinical_fr` | 56.8% | **11.6%** ๐Ÿฅ‡ | 30.9% | 28.0% | 15.3% | ## Clinical / medical de-identification ยท Italian @@ -289,9 +287,9 @@ Corpora in this group: `synth_clinical_it`. A gold PHI span is *leaked* when **no** predicted span overlaps it โ€” 'did we miss a name?' -| Corpus | `anonde-patterns` | `anonde-ner` | `presidio` | `presidio-transformer` | `gliner-py` | `openai-pf` | -|---|---:|---:|---:|---:|---:|---:| -| `synth_clinical_it` | 59.7% | **16.2%** ๐Ÿฅ‡ | 28.5% | โ€“ | 35.2% | 25.1% | +| Corpus | `anonde-patterns` | `anonde-ner` | `presidio` | `gliner-py` | `openai-pf` | +|---|---:|---:|---:|---:|---:| +| `synth_clinical_it` | 59.7% | **16.2%** ๐Ÿฅ‡ | 28.5% | 35.2% | 25.1% | > **Partial coverage** โ€” some engines were benchmarked on a fixed subsample, not every gold doc: > @@ -301,9 +299,9 @@ A gold PHI span is *leaked* when **no** predicted span overlaps it โ€” 'did we m Each leaked span weighted by compliance tier โ€” direct identifiers (PERSON, EMAIL, PHONE, ADDRESS, DOB) = 5, high-stakes IDs (SSN/MRN/IBAN) = 10, quasi-identifiers (LOCATION, ORG, PROFESSION) = 1. Defaults in `label_map.yaml::severity`. Shown only because at least one cell here moves >3pp from raw leak; otherwise the two tables tracked within noise. -| Corpus | `anonde-patterns` | `anonde-ner` | `presidio` | `presidio-transformer` | `gliner-py` | `openai-pf` | -|---|---:|---:|---:|---:|---:|---:| -| `synth_clinical_it` | 55.0% | **15.5%** ๐Ÿฅ‡ | 31.9% | โ€“ | 38.7% | 17.9% | +| Corpus | `anonde-patterns` | `anonde-ner` | `presidio` | `gliner-py` | `openai-pf` | +|---|---:|---:|---:|---:|---:| +| `synth_clinical_it` | 55.0% | **15.5%** ๐Ÿฅ‡ | 31.9% | 38.7% | 17.9% | ## Legal / administrative ยท English @@ -313,9 +311,9 @@ Corpora in this group: `mapa_en`. A gold PHI span is *leaked* when **no** predicted span overlaps it โ€” 'did we miss a name?' -| Corpus | `anonde-patterns` | `anonde-ner` | `presidio` | `presidio-transformer` | `gliner-py` | `openai-pf` | -|---|---:|---:|---:|---:|---:|---:| -| `mapa_en` | 48.9% | 8.4% | **6.2%** ๐Ÿฅ‡ | 13.3% | 10.7% | 76.9% | +| Corpus | `anonde-patterns` | `anonde-ner` | `presidio` | `gliner-py` | `openai-pf` | +|---|---:|---:|---:|---:|---:| +| `mapa_en` | 48.9% | 8.4% | **6.2%** ๐Ÿฅ‡ | 10.7% | 76.9% | > **Partial coverage** โ€” some engines were benchmarked on a fixed subsample, not every gold doc: > @@ -325,9 +323,9 @@ A gold PHI span is *leaked* when **no** predicted span overlaps it โ€” 'did we m Each leaked span weighted by compliance tier โ€” direct identifiers (PERSON, EMAIL, PHONE, ADDRESS, DOB) = 5, high-stakes IDs (SSN/MRN/IBAN) = 10, quasi-identifiers (LOCATION, ORG, PROFESSION) = 1. Defaults in `label_map.yaml::severity`. Shown only because at least one cell here moves >3pp from raw leak; otherwise the two tables tracked within noise. -| Corpus | `anonde-patterns` | `anonde-ner` | `presidio` | `presidio-transformer` | `gliner-py` | `openai-pf` | -|---|---:|---:|---:|---:|---:|---:| -| `mapa_en` | 42.9% | 8.4% | **4.9%** ๐Ÿฅ‡ | 7.5% | 9.8% | 75.4% | +| Corpus | `anonde-patterns` | `anonde-ner` | `presidio` | `gliner-py` | `openai-pf` | +|---|---:|---:|---:|---:|---:| +| `mapa_en` | 42.9% | 8.4% | **4.9%** ๐Ÿฅ‡ | 9.8% | 75.4% | ## Legal / administrative ยท German @@ -337,10 +335,10 @@ Corpora in this group: `legal_de`, `mapa_de`. A gold PHI span is *leaked* when **no** predicted span overlaps it โ€” 'did we miss a name?' -| Corpus | `anonde-patterns` | `anonde-ner` | `presidio` | `presidio-transformer` | `gliner-py` | `openai-pf` | -|---|---:|---:|---:|---:|---:|---:| -| `legal_de` | 6.5% | **0.4%** ๐Ÿฅ‡ | 24.5% | โ€“ | 25.4% | 31.4% | -| `mapa_de` | 51.6% | **6.2%** ๐Ÿฅ‡ | 44.8% | โ€“ | 10.4% | 85.0% | +| Corpus | `anonde-patterns` | `anonde-ner` | `presidio` | `gliner-py` | `openai-pf` | +|---|---:|---:|---:|---:|---:| +| `legal_de` | 6.5% | **0.4%** ๐Ÿฅ‡ | 24.5% | 25.4% | 31.4% | +| `mapa_de` | 51.6% | **6.2%** ๐Ÿฅ‡ | 44.8% | 10.4% | 85.0% | > **Partial coverage** โ€” some engines were benchmarked on a fixed subsample, not every gold doc: > @@ -351,10 +349,10 @@ A gold PHI span is *leaked* when **no** predicted span overlaps it โ€” 'did we m Each leaked span weighted by compliance tier โ€” direct identifiers (PERSON, EMAIL, PHONE, ADDRESS, DOB) = 5, high-stakes IDs (SSN/MRN/IBAN) = 10, quasi-identifiers (LOCATION, ORG, PROFESSION) = 1. Defaults in `label_map.yaml::severity`. Shown only because at least one cell here moves >3pp from raw leak; otherwise the two tables tracked within noise. -| Corpus | `anonde-patterns` | `anonde-ner` | `presidio` | `presidio-transformer` | `gliner-py` | `openai-pf` | -|---|---:|---:|---:|---:|---:|---:| -| `legal_de` | 5.1% | **0.4%** ๐Ÿฅ‡ | 32.9% | โ€“ | 33.3% | 17.0% | -| `mapa_de` | 36.4% | **5.5%** ๐Ÿฅ‡ | 52.0% | โ€“ | 8.4% | 68.8% | +| Corpus | `anonde-patterns` | `anonde-ner` | `presidio` | `gliner-py` | `openai-pf` | +|---|---:|---:|---:|---:|---:| +| `legal_de` | 5.1% | **0.4%** ๐Ÿฅ‡ | 32.9% | 33.3% | 17.0% | +| `mapa_de` | 36.4% | **5.5%** ๐Ÿฅ‡ | 52.0% | 8.4% | 68.8% | ## Legal / administrative ยท Spanish @@ -364,9 +362,9 @@ Corpora in this group: `mapa_es`. A gold PHI span is *leaked* when **no** predicted span overlaps it โ€” 'did we miss a name?' -| Corpus | `anonde-patterns` | `anonde-ner` | `presidio` | `presidio-transformer` | `gliner-py` | `openai-pf` | -|---|---:|---:|---:|---:|---:|---:| -| `mapa_es` | 100.0% | **6.5%** ๐Ÿฅ‡ | 59.1% | โ€“ | 18.3% | 93.8% | +| Corpus | `anonde-patterns` | `anonde-ner` | `presidio` | `gliner-py` | `openai-pf` | +|---|---:|---:|---:|---:|---:| +| `mapa_es` | 100.0% | **6.5%** ๐Ÿฅ‡ | 59.1% | 18.3% | 93.8% | > **Partial coverage** โ€” some engines were benchmarked on a fixed subsample, not every gold doc: > @@ -380,9 +378,9 @@ Corpora in this group: `mapa_fr`. A gold PHI span is *leaked* when **no** predicted span overlaps it โ€” 'did we miss a name?' -| Corpus | `anonde-patterns` | `anonde-ner` | `presidio` | `presidio-transformer` | `gliner-py` | `openai-pf` | -|---|---:|---:|---:|---:|---:|---:| -| `mapa_fr` | 75.2% | **11.4%** ๐Ÿฅ‡ | 44.9% | โ€“ | 16.3% | 95.5% | +| Corpus | `anonde-patterns` | `anonde-ner` | `presidio` | `gliner-py` | `openai-pf` | +|---|---:|---:|---:|---:|---:| +| `mapa_fr` | 75.2% | **11.4%** ๐Ÿฅ‡ | 44.9% | 16.3% | 95.5% | > **Partial coverage** โ€” some engines were benchmarked on a fixed subsample, not every gold doc: > @@ -392,9 +390,9 @@ A gold PHI span is *leaked* when **no** predicted span overlaps it โ€” 'did we m Each leaked span weighted by compliance tier โ€” direct identifiers (PERSON, EMAIL, PHONE, ADDRESS, DOB) = 5, high-stakes IDs (SSN/MRN/IBAN) = 10, quasi-identifiers (LOCATION, ORG, PROFESSION) = 1. Defaults in `label_map.yaml::severity`. Shown only because at least one cell here moves >3pp from raw leak; otherwise the two tables tracked within noise. -| Corpus | `anonde-patterns` | `anonde-ner` | `presidio` | `presidio-transformer` | `gliner-py` | `openai-pf` | -|---|---:|---:|---:|---:|---:|---:| -| `mapa_fr` | 60.3% | **8.9%** ๐Ÿฅ‡ | 53.2% | โ€“ | 11.2% | 98.5% | +| Corpus | `anonde-patterns` | `anonde-ner` | `presidio` | `gliner-py` | `openai-pf` | +|---|---:|---:|---:|---:|---:| +| `mapa_fr` | 60.3% | **8.9%** ๐Ÿฅ‡ | 53.2% | 11.2% | 98.5% | ## Legal / administrative ยท Italian @@ -404,9 +402,9 @@ Corpora in this group: `mapa_it`. A gold PHI span is *leaked* when **no** predicted span overlaps it โ€” 'did we miss a name?' -| Corpus | `anonde-patterns` | `anonde-ner` | `presidio` | `presidio-transformer` | `gliner-py` | `openai-pf` | -|---|---:|---:|---:|---:|---:|---:| -| `mapa_it` | 40.4% | **5.5%** ๐Ÿฅ‡ | 67.1% | โ€“ | 6.2% | 50.0% | +| Corpus | `anonde-patterns` | `anonde-ner` | `presidio` | `gliner-py` | `openai-pf` | +|---|---:|---:|---:|---:|---:| +| `mapa_it` | 40.4% | **5.5%** ๐Ÿฅ‡ | 67.1% | 6.2% | 50.0% | > **Partial coverage** โ€” some engines were benchmarked on a fixed subsample, not every gold doc: > @@ -420,9 +418,9 @@ Corpora in this group: `synth_finance_en`. A gold PHI span is *leaked* when **no** predicted span overlaps it โ€” 'did we miss a name?' -| Corpus | `anonde-patterns` | `anonde-ner` | `presidio` | `presidio-transformer` | `gliner-py` | `openai-pf` | -|---|---:|---:|---:|---:|---:|---:| -| `synth_finance_en` | 21.1% | **2.8%** ๐Ÿฅ‡ | 10.3% | 12.0% | 20.6% | 18.5% | +| Corpus | `anonde-patterns` | `anonde-ner` | `presidio` | `gliner-py` | `openai-pf` | +|---|---:|---:|---:|---:|---:| +| `synth_finance_en` | 21.1% | **2.8%** ๐Ÿฅ‡ | 10.3% | 20.6% | 18.5% | > **Partial coverage** โ€” some engines were benchmarked on a fixed subsample, not every gold doc: > @@ -432,9 +430,9 @@ A gold PHI span is *leaked* when **no** predicted span overlaps it โ€” 'did we m Each leaked span weighted by compliance tier โ€” direct identifiers (PERSON, EMAIL, PHONE, ADDRESS, DOB) = 5, high-stakes IDs (SSN/MRN/IBAN) = 10, quasi-identifiers (LOCATION, ORG, PROFESSION) = 1. Defaults in `label_map.yaml::severity`. Shown only because at least one cell here moves >3pp from raw leak; otherwise the two tables tracked within noise. -| Corpus | `anonde-patterns` | `anonde-ner` | `presidio` | `presidio-transformer` | `gliner-py` | `openai-pf` | -|---|---:|---:|---:|---:|---:|---:| -| `synth_finance_en` | 14.1% | **1.9%** ๐Ÿฅ‡ | 11.9% | 14.2% | 22.7% | 6.0% | +| Corpus | `anonde-patterns` | `anonde-ner` | `presidio` | `gliner-py` | `openai-pf` | +|---|---:|---:|---:|---:|---:| +| `synth_finance_en` | 14.1% | **1.9%** ๐Ÿฅ‡ | 11.9% | 22.7% | 6.0% | ## Retail finance ยท German @@ -444,10 +442,10 @@ Corpora in this group: `finance_de`, `synth_finance_de`. A gold PHI span is *leaked* when **no** predicted span overlaps it โ€” 'did we miss a name?' -| Corpus | `anonde-patterns` | `anonde-ner` | `presidio` | `presidio-transformer` | `gliner-py` | `openai-pf` | -|---|---:|---:|---:|---:|---:|---:| -| `finance_de` | 3.2% | **1.6%** ๐Ÿฅ‡ | 25.9% | โ€“ | 26.2% | 20.8% | -| `synth_finance_de` | **3.8%** ๐Ÿฅ‡ | 4.2% | 21.4% | โ€“ | 17.2% | 22.2% | +| Corpus | `anonde-patterns` | `anonde-ner` | `presidio` | `gliner-py` | `openai-pf` | +|---|---:|---:|---:|---:|---:| +| `finance_de` | 3.2% | **1.6%** ๐Ÿฅ‡ | 25.9% | 26.2% | 20.5% | +| `synth_finance_de` | **3.8%** ๐Ÿฅ‡ | 4.2% | 21.4% | 17.2% | 22.2% | > **Partial coverage** โ€” some engines were benchmarked on a fixed subsample, not every gold doc: > @@ -458,10 +456,10 @@ A gold PHI span is *leaked* when **no** predicted span overlaps it โ€” 'did we m Each leaked span weighted by compliance tier โ€” direct identifiers (PERSON, EMAIL, PHONE, ADDRESS, DOB) = 5, high-stakes IDs (SSN/MRN/IBAN) = 10, quasi-identifiers (LOCATION, ORG, PROFESSION) = 1. Defaults in `label_map.yaml::severity`. Shown only because at least one cell here moves >3pp from raw leak; otherwise the two tables tracked within noise. -| Corpus | `anonde-patterns` | `anonde-ner` | `presidio` | `presidio-transformer` | `gliner-py` | `openai-pf` | -|---|---:|---:|---:|---:|---:|---:| -| `finance_de` | 3.3% | **1.7%** ๐Ÿฅ‡ | 30.2% | โ€“ | 29.2% | 17.1% | -| `synth_finance_de` | 6.2% | **4.9%** ๐Ÿฅ‡ | 23.0% | โ€“ | 17.2% | 9.9% | +| Corpus | `anonde-patterns` | `anonde-ner` | `presidio` | `gliner-py` | `openai-pf` | +|---|---:|---:|---:|---:|---:| +| `finance_de` | 3.3% | **1.7%** ๐Ÿฅ‡ | 30.2% | 29.2% | 16.7% | +| `synth_finance_de` | 6.2% | **4.9%** ๐Ÿฅ‡ | 23.0% | 17.2% | 9.9% | ## Retail finance ยท Spanish @@ -471,9 +469,9 @@ Corpora in this group: `synth_finance_es`. A gold PHI span is *leaked* when **no** predicted span overlaps it โ€” 'did we miss a name?' -| Corpus | `anonde-patterns` | `anonde-ner` | `presidio` | `presidio-transformer` | `gliner-py` | `openai-pf` | -|---|---:|---:|---:|---:|---:|---:| -| `synth_finance_es` | 46.4% | **16.4%** ๐Ÿฅ‡ | 18.5% | โ€“ | 25.4% | 17.4% | +| Corpus | `anonde-patterns` | `anonde-ner` | `presidio` | `gliner-py` | `openai-pf` | +|---|---:|---:|---:|---:|---:| +| `synth_finance_es` | 46.4% | **16.4%** ๐Ÿฅ‡ | 18.5% | 25.4% | 17.6% | > **Partial coverage** โ€” some engines were benchmarked on a fixed subsample, not every gold doc: > @@ -483,9 +481,9 @@ A gold PHI span is *leaked* when **no** predicted span overlaps it โ€” 'did we m Each leaked span weighted by compliance tier โ€” direct identifiers (PERSON, EMAIL, PHONE, ADDRESS, DOB) = 5, high-stakes IDs (SSN/MRN/IBAN) = 10, quasi-identifiers (LOCATION, ORG, PROFESSION) = 1. Defaults in `label_map.yaml::severity`. Shown only because at least one cell here moves >3pp from raw leak; otherwise the two tables tracked within noise. -| Corpus | `anonde-patterns` | `anonde-ner` | `presidio` | `presidio-transformer` | `gliner-py` | `openai-pf` | -|---|---:|---:|---:|---:|---:|---:| -| `synth_finance_es` | 37.2% | 16.2% | 19.9% | โ€“ | 28.8% | **4.7%** ๐Ÿฅ‡ | +| Corpus | `anonde-patterns` | `anonde-ner` | `presidio` | `gliner-py` | `openai-pf` | +|---|---:|---:|---:|---:|---:| +| `synth_finance_es` | 37.2% | 16.2% | 19.9% | 28.8% | **4.8%** ๐Ÿฅ‡ | ## Retail finance ยท French @@ -495,9 +493,9 @@ Corpora in this group: `synth_finance_fr`. A gold PHI span is *leaked* when **no** predicted span overlaps it โ€” 'did we miss a name?' -| Corpus | `anonde-patterns` | `anonde-ner` | `presidio` | `presidio-transformer` | `gliner-py` | `openai-pf` | -|---|---:|---:|---:|---:|---:|---:| -| `synth_finance_fr` | 44.6% | **15.9%** ๐Ÿฅ‡ | 19.4% | โ€“ | 23.9% | 19.3% | +| Corpus | `anonde-patterns` | `anonde-ner` | `presidio` | `gliner-py` | `openai-pf` | +|---|---:|---:|---:|---:|---:| +| `synth_finance_fr` | 44.6% | **15.9%** ๐Ÿฅ‡ | 19.4% | 23.9% | 19.3% | > **Partial coverage** โ€” some engines were benchmarked on a fixed subsample, not every gold doc: > @@ -507,9 +505,9 @@ A gold PHI span is *leaked* when **no** predicted span overlaps it โ€” 'did we m Each leaked span weighted by compliance tier โ€” direct identifiers (PERSON, EMAIL, PHONE, ADDRESS, DOB) = 5, high-stakes IDs (SSN/MRN/IBAN) = 10, quasi-identifiers (LOCATION, ORG, PROFESSION) = 1. Defaults in `label_map.yaml::severity`. Shown only because at least one cell here moves >3pp from raw leak; otherwise the two tables tracked within noise. -| Corpus | `anonde-patterns` | `anonde-ner` | `presidio` | `presidio-transformer` | `gliner-py` | `openai-pf` | -|---|---:|---:|---:|---:|---:|---:| -| `synth_finance_fr` | 35.1% | 13.5% | 21.7% | โ€“ | 26.4% | **6.7%** ๐Ÿฅ‡ | +| Corpus | `anonde-patterns` | `anonde-ner` | `presidio` | `gliner-py` | `openai-pf` | +|---|---:|---:|---:|---:|---:| +| `synth_finance_fr` | 35.1% | 13.5% | 21.7% | 26.4% | **6.7%** ๐Ÿฅ‡ | ## Retail finance ยท Italian @@ -519,9 +517,9 @@ Corpora in this group: `synth_finance_it`. A gold PHI span is *leaked* when **no** predicted span overlaps it โ€” 'did we miss a name?' -| Corpus | `anonde-patterns` | `anonde-ner` | `presidio` | `presidio-transformer` | `gliner-py` | `openai-pf` | -|---|---:|---:|---:|---:|---:|---:| -| `synth_finance_it` | 39.5% | **11.7%** ๐Ÿฅ‡ | 29.6% | โ€“ | 26.4% | 14.1% | +| Corpus | `anonde-patterns` | `anonde-ner` | `presidio` | `gliner-py` | `openai-pf` | +|---|---:|---:|---:|---:|---:| +| `synth_finance_it` | 39.5% | **11.7%** ๐Ÿฅ‡ | 29.6% | 26.4% | 14.1% | > **Partial coverage** โ€” some engines were benchmarked on a fixed subsample, not every gold doc: > @@ -531,9 +529,9 @@ A gold PHI span is *leaked* when **no** predicted span overlaps it โ€” 'did we m Each leaked span weighted by compliance tier โ€” direct identifiers (PERSON, EMAIL, PHONE, ADDRESS, DOB) = 5, high-stakes IDs (SSN/MRN/IBAN) = 10, quasi-identifiers (LOCATION, ORG, PROFESSION) = 1. Defaults in `label_map.yaml::severity`. Shown only because at least one cell here moves >3pp from raw leak; otherwise the two tables tracked within noise. -| Corpus | `anonde-patterns` | `anonde-ner` | `presidio` | `presidio-transformer` | `gliner-py` | `openai-pf` | -|---|---:|---:|---:|---:|---:|---:| -| `synth_finance_it` | 30.5% | 9.7% | 40.5% | โ€“ | 23.8% | **4.1%** ๐Ÿฅ‡ | +| Corpus | `anonde-patterns` | `anonde-ner` | `presidio` | `gliner-py` | `openai-pf` | +|---|---:|---:|---:|---:|---:| +| `synth_finance_it` | 30.5% | 9.7% | 40.5% | 23.8% | **4.1%** ๐Ÿฅ‡ | ## Enterprise logs ยท English @@ -543,9 +541,9 @@ Corpora in this group: `synth_logs`. A gold PHI span is *leaked* when **no** predicted span overlaps it โ€” 'did we miss a name?' -| Corpus | `anonde-patterns` | `anonde-ner` | `presidio` | `presidio-transformer` | `gliner-py` | `openai-pf` | -|---|---:|---:|---:|---:|---:|---:| -| `synth_logs` | 28.9% | **13.2%** ๐Ÿฅ‡ | 31.5% | 37.3% | 73.2% | 15.3% | +| Corpus | `anonde-patterns` | `anonde-ner` | `presidio` | `gliner-py` | `openai-pf` | +|---|---:|---:|---:|---:|---:| +| `synth_logs` | 28.9% | **13.2%** ๐Ÿฅ‡ | 31.5% | 73.2% | 15.2% | > **Partial coverage** โ€” some engines were benchmarked on a fixed subsample, not every gold doc: > @@ -555,9 +553,9 @@ A gold PHI span is *leaked* when **no** predicted span overlaps it โ€” 'did we m Each leaked span weighted by compliance tier โ€” direct identifiers (PERSON, EMAIL, PHONE, ADDRESS, DOB) = 5, high-stakes IDs (SSN/MRN/IBAN) = 10, quasi-identifiers (LOCATION, ORG, PROFESSION) = 1. Defaults in `label_map.yaml::severity`. Shown only because at least one cell here moves >3pp from raw leak; otherwise the two tables tracked within noise. -| Corpus | `anonde-patterns` | `anonde-ner` | `presidio` | `presidio-transformer` | `gliner-py` | `openai-pf` | -|---|---:|---:|---:|---:|---:|---:| -| `synth_logs` | 34.7% | 16.4% | 39.6% | 46.5% | 79.7% | **7.7%** ๐Ÿฅ‡ | +| Corpus | `anonde-patterns` | `anonde-ner` | `presidio` | `gliner-py` | `openai-pf` | +|---|---:|---:|---:|---:|---:| +| `synth_logs` | 34.7% | 16.4% | 39.6% | 79.7% | **7.6%** ๐Ÿฅ‡ | ## General structured PII ยท English @@ -567,9 +565,9 @@ Corpora in this group: `ai4privacy_en`. A gold PHI span is *leaked* when **no** predicted span overlaps it โ€” 'did we miss a name?' -| Corpus | `anonde-patterns` | `anonde-ner` | `presidio` | `presidio-transformer` | `gliner-py` | `openai-pf` | -|---|---:|---:|---:|---:|---:|---:| -| `ai4privacy_en` | 47.8% | **13.0%** ๐Ÿฅ‡ | 56.0% | 57.4% | 35.0% | 15.5% | +| Corpus | `anonde-patterns` | `anonde-ner` | `presidio` | `gliner-py` | `openai-pf` | +|---|---:|---:|---:|---:|---:| +| `ai4privacy_en` | 47.8% | **13.0%** ๐Ÿฅ‡ | 56.0% | 35.0% | 15.5% | > **Partial coverage** โ€” some engines were benchmarked on a fixed subsample, not every gold doc: > @@ -579,9 +577,9 @@ A gold PHI span is *leaked* when **no** predicted span overlaps it โ€” 'did we m Each leaked span weighted by compliance tier โ€” direct identifiers (PERSON, EMAIL, PHONE, ADDRESS, DOB) = 5, high-stakes IDs (SSN/MRN/IBAN) = 10, quasi-identifiers (LOCATION, ORG, PROFESSION) = 1. Defaults in `label_map.yaml::severity`. Shown only because at least one cell here moves >3pp from raw leak; otherwise the two tables tracked within noise. -| Corpus | `anonde-patterns` | `anonde-ner` | `presidio` | `presidio-transformer` | `gliner-py` | `openai-pf` | -|---|---:|---:|---:|---:|---:|---:| -| `ai4privacy_en` | 39.1% | 10.8% | 57.4% | 58.8% | 37.0% | **10.0%** ๐Ÿฅ‡ | +| Corpus | `anonde-patterns` | `anonde-ner` | `presidio` | `gliner-py` | `openai-pf` | +|---|---:|---:|---:|---:|---:| +| `ai4privacy_en` | 39.1% | 10.8% | 57.4% | 37.0% | **10.0%** ๐Ÿฅ‡ | ## General structured PII ยท German @@ -591,9 +589,9 @@ Corpora in this group: `ai4privacy_de`. A gold PHI span is *leaked* when **no** predicted span overlaps it โ€” 'did we miss a name?' -| Corpus | `anonde-patterns` | `anonde-ner` | `presidio` | `presidio-transformer` | `gliner-py` | `openai-pf` | -|---|---:|---:|---:|---:|---:|---:| -| `ai4privacy_de` | 59.8% | **10.7%** ๐Ÿฅ‡ | 58.4% | โ€“ | 33.0% | 23.6% | +| Corpus | `anonde-patterns` | `anonde-ner` | `presidio` | `gliner-py` | `openai-pf` | +|---|---:|---:|---:|---:|---:| +| `ai4privacy_de` | 59.8% | **10.7%** ๐Ÿฅ‡ | 58.4% | 33.0% | 23.6% | > **Partial coverage** โ€” some engines were benchmarked on a fixed subsample, not every gold doc: > @@ -603,9 +601,9 @@ A gold PHI span is *leaked* when **no** predicted span overlaps it โ€” 'did we m Each leaked span weighted by compliance tier โ€” direct identifiers (PERSON, EMAIL, PHONE, ADDRESS, DOB) = 5, high-stakes IDs (SSN/MRN/IBAN) = 10, quasi-identifiers (LOCATION, ORG, PROFESSION) = 1. Defaults in `label_map.yaml::severity`. Shown only because at least one cell here moves >3pp from raw leak; otherwise the two tables tracked within noise. -| Corpus | `anonde-patterns` | `anonde-ner` | `presidio` | `presidio-transformer` | `gliner-py` | `openai-pf` | -|---|---:|---:|---:|---:|---:|---:| -| `ai4privacy_de` | 63.9% | **9.7%** ๐Ÿฅ‡ | 62.3% | โ€“ | 34.7% | 16.3% | +| Corpus | `anonde-patterns` | `anonde-ner` | `presidio` | `gliner-py` | `openai-pf` | +|---|---:|---:|---:|---:|---:| +| `ai4privacy_de` | 63.9% | **9.7%** ๐Ÿฅ‡ | 62.3% | 34.7% | 16.3% | ## General structured PII ยท Spanish @@ -615,9 +613,9 @@ Corpora in this group: `ai4privacy_es`. A gold PHI span is *leaked* when **no** predicted span overlaps it โ€” 'did we miss a name?' -| Corpus | `anonde-patterns` | `anonde-ner` | `presidio` | `presidio-transformer` | `gliner-py` | `openai-pf` | -|---|---:|---:|---:|---:|---:|---:| -| `ai4privacy_es` | 68.9% | **14.2%** ๐Ÿฅ‡ | 61.0% | โ€“ | 34.6% | 17.5% | +| Corpus | `anonde-patterns` | `anonde-ner` | `presidio` | `gliner-py` | `openai-pf` | +|---|---:|---:|---:|---:|---:| +| `ai4privacy_es` | 68.9% | **14.2%** ๐Ÿฅ‡ | 61.0% | 34.6% | 17.5% | > **Partial coverage** โ€” some engines were benchmarked on a fixed subsample, not every gold doc: > @@ -627,9 +625,9 @@ A gold PHI span is *leaked* when **no** predicted span overlaps it โ€” 'did we m Each leaked span weighted by compliance tier โ€” direct identifiers (PERSON, EMAIL, PHONE, ADDRESS, DOB) = 5, high-stakes IDs (SSN/MRN/IBAN) = 10, quasi-identifiers (LOCATION, ORG, PROFESSION) = 1. Defaults in `label_map.yaml::severity`. Shown only because at least one cell here moves >3pp from raw leak; otherwise the two tables tracked within noise. -| Corpus | `anonde-patterns` | `anonde-ner` | `presidio` | `presidio-transformer` | `gliner-py` | `openai-pf` | -|---|---:|---:|---:|---:|---:|---:| -| `ai4privacy_es` | 68.7% | 11.2% | 65.1% | โ€“ | 35.2% | **7.6%** ๐Ÿฅ‡ | +| Corpus | `anonde-patterns` | `anonde-ner` | `presidio` | `gliner-py` | `openai-pf` | +|---|---:|---:|---:|---:|---:| +| `ai4privacy_es` | 68.7% | 11.2% | 65.1% | 35.2% | **7.6%** ๐Ÿฅ‡ | ## General structured PII ยท French @@ -639,9 +637,9 @@ Corpora in this group: `ai4privacy_fr`. A gold PHI span is *leaked* when **no** predicted span overlaps it โ€” 'did we miss a name?' -| Corpus | `anonde-patterns` | `anonde-ner` | `presidio` | `presidio-transformer` | `gliner-py` | `openai-pf` | -|---|---:|---:|---:|---:|---:|---:| -| `ai4privacy_fr` | 68.9% | **14.2%** ๐Ÿฅ‡ | 54.0% | โ€“ | 34.9% | 16.3% | +| Corpus | `anonde-patterns` | `anonde-ner` | `presidio` | `gliner-py` | `openai-pf` | +|---|---:|---:|---:|---:|---:| +| `ai4privacy_fr` | 68.9% | **14.2%** ๐Ÿฅ‡ | 54.0% | 34.9% | 16.3% | > **Partial coverage** โ€” some engines were benchmarked on a fixed subsample, not every gold doc: > @@ -651,9 +649,9 @@ A gold PHI span is *leaked* when **no** predicted span overlaps it โ€” 'did we m Each leaked span weighted by compliance tier โ€” direct identifiers (PERSON, EMAIL, PHONE, ADDRESS, DOB) = 5, high-stakes IDs (SSN/MRN/IBAN) = 10, quasi-identifiers (LOCATION, ORG, PROFESSION) = 1. Defaults in `label_map.yaml::severity`. Shown only because at least one cell here moves >3pp from raw leak; otherwise the two tables tracked within noise. -| Corpus | `anonde-patterns` | `anonde-ner` | `presidio` | `presidio-transformer` | `gliner-py` | `openai-pf` | -|---|---:|---:|---:|---:|---:|---:| -| `ai4privacy_fr` | 67.1% | 11.7% | 56.7% | โ€“ | 35.8% | **9.3%** ๐Ÿฅ‡ | +| Corpus | `anonde-patterns` | `anonde-ner` | `presidio` | `gliner-py` | `openai-pf` | +|---|---:|---:|---:|---:|---:| +| `ai4privacy_fr` | 67.1% | 11.7% | 56.7% | 35.8% | **9.3%** ๐Ÿฅ‡ | ## General structured PII ยท Italian @@ -663,9 +661,9 @@ Corpora in this group: `ai4privacy_it`. A gold PHI span is *leaked* when **no** predicted span overlaps it โ€” 'did we miss a name?' -| Corpus | `anonde-patterns` | `anonde-ner` | `presidio` | `presidio-transformer` | `gliner-py` | `openai-pf` | -|---|---:|---:|---:|---:|---:|---:| -| `ai4privacy_it` | 63.9% | **15.0%** ๐Ÿฅ‡ | 59.6% | โ€“ | 36.0% | 21.2% | +| Corpus | `anonde-patterns` | `anonde-ner` | `presidio` | `gliner-py` | `openai-pf` | +|---|---:|---:|---:|---:|---:| +| `ai4privacy_it` | 63.9% | **15.0%** ๐Ÿฅ‡ | 59.6% | 36.0% | 21.2% | > **Partial coverage** โ€” some engines were benchmarked on a fixed subsample, not every gold doc: > @@ -675,9 +673,9 @@ A gold PHI span is *leaked* when **no** predicted span overlaps it โ€” 'did we m Each leaked span weighted by compliance tier โ€” direct identifiers (PERSON, EMAIL, PHONE, ADDRESS, DOB) = 5, high-stakes IDs (SSN/MRN/IBAN) = 10, quasi-identifiers (LOCATION, ORG, PROFESSION) = 1. Defaults in `label_map.yaml::severity`. Shown only because at least one cell here moves >3pp from raw leak; otherwise the two tables tracked within noise. -| Corpus | `anonde-patterns` | `anonde-ner` | `presidio` | `presidio-transformer` | `gliner-py` | `openai-pf` | -|---|---:|---:|---:|---:|---:|---:| -| `ai4privacy_it` | 58.9% | **11.5%** ๐Ÿฅ‡ | 62.4% | โ€“ | 36.6% | 13.6% | +| Corpus | `anonde-patterns` | `anonde-ner` | `presidio` | `gliner-py` | `openai-pf` | +|---|---:|---:|---:|---:|---:| +| `ai4privacy_it` | 58.9% | **11.5%** ๐Ÿฅ‡ | 62.4% | 36.6% | 13.6% | ## Academic NER (newswire / social) ยท English @@ -687,10 +685,10 @@ Corpora in this group: `conll2003_en`, `wnut_17`. A gold PHI span is *leaked* when **no** predicted span overlaps it โ€” 'did we miss a name?' -| Corpus | `anonde-patterns` | `anonde-ner` | `presidio` | `presidio-transformer` | `gliner-py` | `openai-pf` | -|---|---:|---:|---:|---:|---:|---:| -| `conll2003_en` | 96.7% | **4.7%** ๐Ÿฅ‡ | 6.7% | 12.2% | 6.0% | 64.0% | -| `wnut_17` | 80.9% | **11.2%** ๐Ÿฅ‡ | 43.1% | 26.6% | 25.5% | 82.1% | +| Corpus | `anonde-patterns` | `anonde-ner` | `presidio` | `gliner-py` | `openai-pf` | +|---|---:|---:|---:|---:|---:| +| `conll2003_en` | 96.7% | **4.7%** ๐Ÿฅ‡ | 6.7% | 6.0% | 64.0% | +| `wnut_17` | 80.9% | **11.2%** ๐Ÿฅ‡ | 43.1% | 25.5% | 82.1% | > **Partial coverage** โ€” some engines were benchmarked on a fixed subsample, not every gold doc: > @@ -701,10 +699,10 @@ A gold PHI span is *leaked* when **no** predicted span overlaps it โ€” 'did we m Each leaked span weighted by compliance tier โ€” direct identifiers (PERSON, EMAIL, PHONE, ADDRESS, DOB) = 5, high-stakes IDs (SSN/MRN/IBAN) = 10, quasi-identifiers (LOCATION, ORG, PROFESSION) = 1. Defaults in `label_map.yaml::severity`. Shown only because at least one cell here moves >3pp from raw leak; otherwise the two tables tracked within noise. -| Corpus | `anonde-patterns` | `anonde-ner` | `presidio` | `presidio-transformer` | `gliner-py` | `openai-pf` | -|---|---:|---:|---:|---:|---:|---:| -| `conll2003_en` | 97.0% | **5.2%** ๐Ÿฅ‡ | 5.3% | 6.2% | 6.9% | 47.9% | -| `wnut_17` | 74.6% | **6.7%** ๐Ÿฅ‡ | 42.2% | 21.4% | 25.4% | 61.5% | +| Corpus | `anonde-patterns` | `anonde-ner` | `presidio` | `gliner-py` | `openai-pf` | +|---|---:|---:|---:|---:|---:| +| `conll2003_en` | 97.0% | **5.2%** ๐Ÿฅ‡ | 5.3% | 6.9% | 47.9% | +| `wnut_17` | 74.6% | **6.7%** ๐Ÿฅ‡ | 42.2% | 25.4% | 61.5% | ## Academic NER (newswire / social) ยท German @@ -714,10 +712,10 @@ Corpora in this group: `wikiann_de`, `germeval_14`, `conll2003_de`. A gold PHI span is *leaked* when **no** predicted span overlaps it โ€” 'did we miss a name?' -| Corpus | `anonde-patterns` | `anonde-ner` | `presidio` | `presidio-transformer` | `gliner-py` | `openai-pf` | -|---|---:|---:|---:|---:|---:|---:| -| `wikiann_de` | 41.9% | **3.0%** ๐Ÿฅ‡ | 16.7% | โ€“ | 12.8% | 59.6% | -| `germeval_14` | 55.0% | **9.3%** ๐Ÿฅ‡ | 22.0% | โ€“ | 18.9% | 94.6% | +| Corpus | `anonde-patterns` | `anonde-ner` | `presidio` | `gliner-py` | `openai-pf` | +|---|---:|---:|---:|---:|---:| +| `wikiann_de` | 41.9% | **3.0%** ๐Ÿฅ‡ | 16.7% | 12.8% | 59.6% | +| `germeval_14` | 55.0% | **9.3%** ๐Ÿฅ‡ | 22.0% | 18.9% | 94.6% | > **Partial coverage** โ€” some engines were benchmarked on a fixed subsample, not every gold doc: > @@ -728,10 +726,10 @@ A gold PHI span is *leaked* when **no** predicted span overlaps it โ€” 'did we m Each leaked span weighted by compliance tier โ€” direct identifiers (PERSON, EMAIL, PHONE, ADDRESS, DOB) = 5, high-stakes IDs (SSN/MRN/IBAN) = 10, quasi-identifiers (LOCATION, ORG, PROFESSION) = 1. Defaults in `label_map.yaml::severity`. Shown only because at least one cell here moves >3pp from raw leak; otherwise the two tables tracked within noise. -| Corpus | `anonde-patterns` | `anonde-ner` | `presidio` | `presidio-transformer` | `gliner-py` | `openai-pf` | -|---|---:|---:|---:|---:|---:|---:| -| `wikiann_de` | 30.3% | **1.9%** ๐Ÿฅ‡ | 6.9% | โ€“ | 10.7% | 44.3% | -| `germeval_14` | 47.5% | **6.2%** ๐Ÿฅ‡ | 15.8% | โ€“ | 16.1% | 88.8% | +| Corpus | `anonde-patterns` | `anonde-ner` | `presidio` | `gliner-py` | `openai-pf` | +|---|---:|---:|---:|---:|---:| +| `wikiann_de` | 30.3% | **1.9%** ๐Ÿฅ‡ | 6.9% | 10.7% | 44.3% | +| `germeval_14` | 47.5% | **6.2%** ๐Ÿฅ‡ | 15.8% | 16.1% | 88.8% | ## Adversarial / out-of-distribution ยท German @@ -741,9 +739,9 @@ Corpora in this group: `adversarial_de`. A gold PHI span is *leaked* when **no** predicted span overlaps it โ€” 'did we miss a name?' -| Corpus | `anonde-patterns` | `anonde-ner` | `presidio` | `presidio-transformer` | `gliner-py` | `openai-pf` | -|---|---:|---:|---:|---:|---:|---:| -| `adversarial_de` | 12.7% | **8.1%** ๐Ÿฅ‡ | 37.6% | โ€“ | 43.4% | 32.0% | +| Corpus | `anonde-patterns` | `anonde-ner` | `presidio` | `gliner-py` | `openai-pf` | +|---|---:|---:|---:|---:|---:| +| `adversarial_de` | 12.6% | **8.0%** ๐Ÿฅ‡ | 37.5% | 43.3% | 32.0% | > **Partial coverage** โ€” some engines were benchmarked on a fixed subsample, not every gold doc: > @@ -753,46 +751,46 @@ A gold PHI span is *leaked* when **no** predicted span overlaps it โ€” 'did we m Each leaked span weighted by compliance tier โ€” direct identifiers (PERSON, EMAIL, PHONE, ADDRESS, DOB) = 5, high-stakes IDs (SSN/MRN/IBAN) = 10, quasi-identifiers (LOCATION, ORG, PROFESSION) = 1. Defaults in `label_map.yaml::severity`. Shown only because at least one cell here moves >3pp from raw leak; otherwise the two tables tracked within noise. -| Corpus | `anonde-patterns` | `anonde-ner` | `presidio` | `presidio-transformer` | `gliner-py` | `openai-pf` | -|---|---:|---:|---:|---:|---:|---:| -| `adversarial_de` | 12.2% | **8.1%** ๐Ÿฅ‡ | 40.8% | โ€“ | 44.0% | 27.7% | +| Corpus | `anonde-patterns` | `anonde-ner` | `presidio` | `gliner-py` | `openai-pf` | +|---|---:|---:|---:|---:|---:| +| `adversarial_de` | 12.1% | **8.0%** ๐Ÿฅ‡ | 40.7% | 43.8% | 27.7% | ## Latency ยท per-document p50 / p95 Wall-clock per `engine.Analyze(doc)` call. p50 = steady-state, p95 = tail (the SLO knob). Mean + p99 in `results_matrix.csv`. One table across every corpus โ€” latency tracks corpus length, not domain or language. -| Corpus | `anonde-patterns` p50 / p95 | `anonde-ner` p50 / p95 | `presidio` p50 / p95 | `presidio-transformer` p50 / p95 | `gliner-py` p50 / p95 | `openai-pf` p50 / p95 | -|---|---:|---:|---:|---:|---:|---:| -| `synth_clinical_en` | 3 ms / 5 ms | 467 ms / 741 ms | 41 ms / 56 ms | 477 ms / 636 ms | 528 ms / 669 ms | 1.5 s / 2.0 s | -| `openmed` | 9 ms / 28 ms | 1.3 s / 4.4 s | 135 ms / 336 ms | โ€“ | 2.1 s / 9.9 s | 7.9 s / 22.2 s | -| `synth_clinical` | 3 ms / 5 ms | 320 ms / 512 ms | 30 ms / 39 ms | โ€“ | 415 ms / 573 ms | 745 ms / 860 ms | -| `pharmaconer_es` | 4 ms / 8 ms | 1.0 s / 2.3 s | 64 ms / 151 ms | โ€“ | 1.2 s / 3.2 s | 4.1 s / 8.9 s | -| `meddocan_es` | 5 ms / 9 ms | 1.3 s / 2.5 s | 87 ms / 171 ms | โ€“ | 1.6 s / 3.6 s | 5.0 s / 8.3 s | -| `synth_clinical_fr` | 2 ms / 3 ms | 613 ms / 805 ms | 48 ms / 65 ms | โ€“ | 653 ms / 894 ms | 1.8 s / 2.6 s | -| `synth_clinical_it` | 3 ms / 4 ms | 469 ms / 796 ms | 48 ms / 66 ms | โ€“ | 663 ms / 953 ms | 2.1 s / 2.9 s | -| `mapa_en` | 1 ms / 2 ms | 104 ms / 260 ms | 8 ms / 23 ms | 69 ms / 301 ms | 173 ms / 333 ms | 208 ms / 491 ms | -| `legal_de` | 2 ms / 3 ms | 383 ms / 477 ms | 36 ms / 51 ms | โ€“ | 471 ms / 594 ms | 1.2 s / 1.6 s | -| `mapa_de` | 1 ms / 1 ms | 127 ms / 246 ms | 10 ms / 18 ms | โ€“ | 223 ms / 347 ms | 304 ms / 593 ms | -| `mapa_es` | 0 ms / 1 ms | 126 ms / 335 ms | 10 ms / 28 ms | โ€“ | 212 ms / 413 ms | 354 ms / 798 ms | -| `mapa_fr` | 0 ms / 1 ms | 127 ms / 274 ms | 11 ms / 28 ms | โ€“ | 214 ms / 359 ms | 393 ms / 772 ms | -| `mapa_it` | 0 ms / 1 ms | 122 ms / 228 ms | 10 ms / 18 ms | โ€“ | 207 ms / 314 ms | 383 ms / 682 ms | -| `synth_finance_en` | 2 ms / 3 ms | 289 ms / 479 ms | 31 ms / 55 ms | 335 ms / 642 ms | 368 ms / 577 ms | 786 ms / 958 ms | -| `finance_de` | 3 ms / 4 ms | 305 ms / 453 ms | 31 ms / 51 ms | โ€“ | 350 ms / 520 ms | 842 ms / 953 ms | -| `synth_finance_de` | 2 ms / 3 ms | 319 ms / 482 ms | 28 ms / 57 ms | โ€“ | 401 ms / 641 ms | 927 ms / 1.2 s | -| `synth_finance_es` | 1 ms / 2 ms | 341 ms / 493 ms | 30 ms / 55 ms | โ€“ | 430 ms / 669 ms | 892 ms / 1.2 s | -| `synth_finance_fr` | 1 ms / 2 ms | 356 ms / 495 ms | 35 ms / 60 ms | โ€“ | 446 ms / 680 ms | 979 ms / 1.2 s | -| `synth_finance_it` | 1 ms / 3 ms | 355 ms / 612 ms | 31 ms / 55 ms | โ€“ | 442 ms / 714 ms | 1.0 s / 1.3 s | -| `synth_logs` | 4 ms / 8 ms | 745 ms / 1.5 s | 61 ms / 150 ms | 1.1 s / 2.5 s | 1.2 s / 3.2 s | 7.6 s / 10.8 s | -| `ai4privacy_en` | 1 ms / 2 ms | 218 ms / 324 ms | 22 ms / 31 ms | 281 ms / 337 ms | 312 ms / 435 ms | 620 ms / 837 ms | -| `ai4privacy_de` | 1 ms / 2 ms | 234 ms / 336 ms | 20 ms / 29 ms | โ€“ | 330 ms / 441 ms | 689 ms / 908 ms | -| `ai4privacy_es` | 1 ms / 1 ms | 233 ms / 340 ms | 19 ms / 27 ms | โ€“ | 334 ms / 448 ms | 678 ms / 798 ms | -| `ai4privacy_fr` | 1 ms / 1 ms | 262 ms / 358 ms | 26 ms / 37 ms | โ€“ | 391 ms / 510 ms | 614 ms / 805 ms | -| `ai4privacy_it` | 1 ms / 1 ms | 237 ms / 340 ms | 20 ms / 29 ms | โ€“ | 332 ms / 448 ms | 716 ms / 933 ms | -| `conll2003_en` | 0 ms / 1 ms | 68 ms / 122 ms | 7 ms / 13 ms | 54 ms / 87 ms | 149 ms / 201 ms | 173 ms / 349 ms | -| `wnut_17` | 0 ms / 1 ms | 93 ms / 167 ms | 7 ms / 13 ms | 69 ms / 125 ms | 190 ms / 269 ms | 226 ms / 352 ms | -| `wikiann_de` | 0 ms / 1 ms | 66 ms / 104 ms | 5 ms / 9 ms | โ€“ | 152 ms / 190 ms | 128 ms / 302 ms | -| `germeval_14` | 0 ms / 1 ms | 94 ms / 142 ms | 7 ms / 11 ms | โ€“ | 190 ms / 240 ms | 199 ms / 339 ms | -| `adversarial_de` | 3 ms / 4 ms | 463 ms / 691 ms | 45 ms / 59 ms | โ€“ | 675 ms / 1.1 s | 2.9 s / 4.2 s | +| Corpus | `anonde-patterns` p50 / p95 | `anonde-ner` p50 / p95 | `presidio` p50 / p95 | `gliner-py` p50 / p95 | `openai-pf` p50 / p95 | +|---|---:|---:|---:|---:|---:| +| `synth_clinical_en` | 3 ms / 4 ms | 474 ms / 748 ms | 45 ms / 59 ms | 522 ms / 657 ms | 1.5 s / 2.0 s | +| `openmed` | 9 ms / 28 ms | 1.3 s / 4.5 s | 126 ms / 330 ms | 2.0 s / 9.5 s | 7.8 s / 22.4 s | +| `synth_clinical` | 3 ms / 4 ms | 444 ms / 714 ms | 44 ms / 55 ms | 590 ms / 797 ms | 1.6 s / 2.3 s | +| `pharmaconer_es` | 4 ms / 8 ms | 992 ms / 2.2 s | 61 ms / 144 ms | 1.2 s / 3.0 s | 4.0 s / 8.6 s | +| `meddocan_es` | 5 ms / 9 ms | 1.4 s / 2.6 s | 91 ms / 176 ms | 1.7 s / 3.8 s | 4.5 s / 7.3 s | +| `synth_clinical_fr` | 2 ms / 3 ms | 680 ms / 886 ms | 51 ms / 69 ms | 736 ms / 1.0 s | 1.7 s / 2.4 s | +| `synth_clinical_it` | 2 ms / 3 ms | 457 ms / 792 ms | 45 ms / 63 ms | 635 ms / 887 ms | 2.1 s / 2.9 s | +| `mapa_en` | 1 ms / 2 ms | 113 ms / 296 ms | 10 ms / 29 ms | 190 ms / 361 ms | 308 ms / 950 ms | +| `legal_de` | 2 ms / 3 ms | 413 ms / 516 ms | 37 ms / 53 ms | 524 ms / 646 ms | 1.1 s / 1.5 s | +| `mapa_de` | 1 ms / 1 ms | 116 ms / 225 ms | 10 ms / 18 ms | 196 ms / 302 ms | 338 ms / 629 ms | +| `mapa_es` | 1 ms / 1 ms | 130 ms / 343 ms | 11 ms / 29 ms | 212 ms / 416 ms | 347 ms / 797 ms | +| `mapa_fr` | 0 ms / 1 ms | 126 ms / 277 ms | 12 ms / 28 ms | 210 ms / 348 ms | 393 ms / 771 ms | +| `mapa_it` | 0 ms / 1 ms | 132 ms / 253 ms | 10 ms / 18 ms | 226 ms / 349 ms | 339 ms / 612 ms | +| `synth_finance_en` | 2 ms / 3 ms | 288 ms / 482 ms | 32 ms / 57 ms | 361 ms / 566 ms | 769 ms / 956 ms | +| `finance_de` | 1 ms / 2 ms | 202 ms / 283 ms | 27 ms / 42 ms | 421 ms / 616 ms | 501 ms / 572 ms | +| `synth_finance_de` | 2 ms / 3 ms | 318 ms / 479 ms | 28 ms / 57 ms | 387 ms / 606 ms | 909 ms / 1.2 s | +| `synth_finance_es` | 1 ms / 2 ms | 222 ms / 332 ms | 20 ms / 38 ms | 263 ms / 422 ms | 433 ms / 494 ms | +| `synth_finance_fr` | 1 ms / 2 ms | 353 ms / 503 ms | 34 ms / 61 ms | 432 ms / 645 ms | 990 ms / 1.3 s | +| `synth_finance_it` | 1 ms / 2 ms | 392 ms / 543 ms | 31 ms / 54 ms | 491 ms / 765 ms | 924 ms / 1.2 s | +| `synth_logs` | 2 ms / 5 ms | 299 ms / 590 ms | 33 ms / 81 ms | 834 ms / 2.3 s | 1.0 s / 1.5 s | +| `ai4privacy_en` | 1 ms / 2 ms | 218 ms / 325 ms | 23 ms / 31 ms | 310 ms / 424 ms | 634 ms / 840 ms | +| `ai4privacy_de` | 1 ms / 2 ms | 258 ms / 368 ms | 22 ms / 31 ms | 373 ms / 498 ms | 650 ms / 849 ms | +| `ai4privacy_es` | 1 ms / 1 ms | 232 ms / 338 ms | 19 ms / 26 ms | 319 ms / 429 ms | 706 ms / 825 ms | +| `ai4privacy_fr` | 1 ms / 1 ms | 238 ms / 327 ms | 25 ms / 35 ms | 324 ms / 415 ms | 647 ms / 853 ms | +| `ai4privacy_it` | 1 ms / 1 ms | 237 ms / 342 ms | 20 ms / 28 ms | 327 ms / 436 ms | 746 ms / 958 ms | +| `conll2003_en` | 0 ms / 1 ms | 50 ms / 85 ms | 5 ms / 10 ms | 98 ms / 131 ms | 157 ms / 289 ms | +| `wnut_17` | 0 ms / 1 ms | 80 ms / 146 ms | 6 ms / 13 ms | 165 ms / 230 ms | 179 ms / 258 ms | +| `wikiann_de` | 0 ms / 1 ms | 67 ms / 105 ms | 5 ms / 9 ms | 153 ms / 190 ms | 130 ms / 309 ms | +| `germeval_14` | 0 ms / 1 ms | 86 ms / 131 ms | 7 ms / 11 ms | 171 ms / 213 ms | 226 ms / 357 ms | +| `adversarial_de` | 3 ms / 4 ms | 467 ms / 693 ms | 44 ms / 59 ms | 654 ms / 1.1 s | 2.9 s / 4.1 s |
Cost reference ยท USD per million characters @@ -874,4 +872,4 @@ in this matrix: `openmed` (GraSCCo PHI), `synth_clinical`,
--- -*Generated by `bench/scoring/render_matrix.py` over 157 cells. Full per-entity-type breakdown in `results_matrix.csv`.* +*Generated by `bench/scoring/render_matrix.py` over 150 cells. Full per-entity-type breakdown in `results_matrix.csv`.*