NVIDIA-Medtech · medatasci · Jul 16, 2026
diff --git a/skills/report-anonymization/BENCHMARK.md b/skills/report-anonymization/BENCHMARK.md
@@ -0,0 +1,83 @@
+<!--
+SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+SPDX-License-Identifier: Apache-2.0
+-->
+
+# Benchmark — report-anonymization
+
+Engineering behavior benchmark for the report-anonymization skill. This is an
+integration/quality signal, not a clinical or regulatory claim.
+
+## Task
+
+De-identify English radiology reports (replace patient/doctor names, MRNs,
+dates, accession numbers, and institutions with bracketed role tokens via NeMo
+Anonymizer Redact) and emit a schema-valid `anonymization_summary` plus an
+`anonymized_reports.csv`.
+
+## With-skill vs without-skill
+
+The with-vs-without protocol compares `LLM + report-anonymization/SKILL.md`
+against `LLM + upstream NeMo Anonymizer docs` on the same task and the same
+staged input (the MR-RATE `batch00_reports_w_PHI.csv` reports). Both arms run
+the real NeMo Anonymizer pipeline for tier-5 execution; they differ only in the
+documentation the agent may read.
+
+- **Without the skill**, an agent must discover from the upstream README/skill
+  how to wire `AnonymizerInput` / `Detect` (strict label mode) / `Redact`,
+  which text column to use, and where to write output — and typically improvises
+  an unaudited script with no machine-readable summary.
+- **With the skill**, the agent gets one documented entrypoint that owns the
+  strict PHI label set, output shaping (`study_uid,report`), a schema-gated JSON
+  summary, per-stage entity telemetry, and the residual-leak audit.
+
+## Results
+
+Full run: single-shot, no-repair; 1 repeat; **100 reports** staged from the
+MR-RATE `batch00_reports_w_PHI.csv`; two build.nvidia.com backends
+(Nemotron-3-Super-120B, GPT-OSS-120B). Pass criterion: **any residual PHI escape
+is a fail**, where escapes are counted by an **LLM-as-judge** (GPT-OSS-120B) that
+re-reads each anonymized report. Produced-output (tier-5 completion) is reported
+separately.
+
+| Arm | Produced output | Residual PHI escapes (LLM judge) | Rows fully redacted | Pass (0 escapes) |
+|---|:--:|--:|:--:|:--:|
+| **with skill** (`report-anonymization/SKILL.md`) | 2/2 | **21** | **197/200** | 0/2 |
+| without skill (upstream NeMo README) | 2/2 | 227 | 85/200 | 0/2 |
+
+- **Redaction quality is the decisive gap (~11x).** With the skill, agents left **21** residual PHI escapes and fully redacted **197/200** rows (98.5%); reading only the upstream README they left **227** escapes and fully redacted just **85/200** rows (42.5%). The skill's strict GLiNER label set drives detection; the upstream-default path leaves pervasive residual fragments (mostly name middle initials, plus some dates/institutions).
+- **Neither arm clears the strict zero-escape bar at 100-row scale** (both 0/2 pass): the with-skill misses were a handful of names/one institution GLiNER did not propose (e.g. `Mercy General Hospital`, `Raoul`). The escape counts — not the pass/fail — are the primary signal.
+- **Speed:** mean tier-5 exec ~293s with-skill vs ~343s without on 100 rows (the upstream-default path does far more augmentation; gpt-oss/without ran ~362s).
+- **10-row pilot with a third `unaided` arm** (plain "redact this CSV" request, no doc) is preserved in git history of the report: unaided left 47 escapes / 0 rows clean and one backend produced no output — the hardest baseline.
+
+Full per-backend/per-arm detail, per-row judge escape counts, generated commands,
+token profiling, and the five-tier grade are in the checked-in report:
+
+- [`docs/anonymization-with-vs-without-experiment.md`](../../docs/anonymization-with-vs-without-experiment.md)
+
+Reproduce (from the `medical-AI-skills` catalog root):
+
+```bash
+export NVIDIA_API_KEY="nvapi-..."
+# with + without arms
+python -m tools.curation_eval.anon_experiment \
+  --backends nemotron120-remote "gptoss=https://integrate.api.nvidia.com/v1=openai/gpt-oss-120b=NVIDIA_API_KEY" \
+  --judge "gptoss=https://integrate.api.nvidia.com/v1=openai/gpt-oss-120b=NVIDIA_API_KEY" \
+  --repeats 1 --limit 10 --timeout 120 \
+  --input <path>/batch00_reports_w_PHI.csv
+# add the unaided arm, folding in a prior study, judging all arms
+python -m tools.curation_eval.anon_experiment --arms unaided \
+  --backends nemotron120-remote "gptoss=…=NVIDIA_API_KEY" \
+  --judge "gptoss=…=NVIDIA_API_KEY" --merge runs/curation_eval/anon/<prior_study> \
+  --repeats 1 --limit 10 --timeout 120 --input <path>/batch00_reports_w_PHI.csv
+```
+
+Scale up with `--limit 100` (and more `--repeats`) when the run budget allows.
+
+## Gaps
+
+- Detection runs on remote LLMs (build.nvidia.com); throughput and exact leak
+  counts depend on the model and rate limits and are recorded per run, not
+  pinned here.
+- Grading measures task completion (a contract-valid anonymized artifact) and a
+  deterministic residual-PHI heuristic, not clinical de-identification quality.
diff --git a/skills/report-anonymization/README.md b/skills/report-anonymization/README.md
@@ -0,0 +1,128 @@
+<!--
+SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+SPDX-License-Identifier: Apache-2.0
+-->
+
+# report-anonymization
+
+De-identify English radiology reports by replacing PHI entities detected by
+[NeMo Anonymizer](https://github.com/NVIDIA-NeMo/Anonymizer) (GLiNER-PII
+detection + LLM augmentation/validation) with bracketed role tokens
+(`[PATIENT]`, `[DOCTOR]`, `[DATE]`, ...). This is **MR-RATE
+`reports_preprocessing` stage 01**, packaged as an agent-callable Medical AI
+Skill around the upstream `nemo-anonymizer` package.
+
+> **Engineering / research use only.** This skill is **not** a regulatory
+> de-identifier and does not guarantee removal of all PHI. It is not for
+> clinical deployment, autonomous diagnosis, or patient-facing use. Review
+> output before sharing data.
+
+## What it does
+
+- Calls the documented `Anonymizer()` Python API in Replace mode (`Redact`,
+  `format_template="[{label}]"`) with a strict GLiNER label set for report PHI.
+- Emits an `anonymization_summary` JSON on stdout (progress logs go to stderr)
+  and writes `anonymized_reports.csv` (`study_uid,report`) plus a detailed
+  `run_report.json` to the output directory. Preview mode also writes
+  `preview.parquet` (the full pipeline trace).
+- Reports per-stage entity counts, timings, throughput, a tiktoken input-token
+  estimate, and an informational `residual_phi_leak` verbatim check.
+
+| I/O | Detail |
+|---|---|
+| Input | `reports_csv` — CSV with `study_uid` + `report_w_PHI` columns |
+| Output (stdout) | `anonymization_summary` JSON (validated against `validators/output_schema.json`) |
+| Output (files) | `<output-dir>/anonymized_reports.csv`, `<output-dir>/run_report.json`, `<output-dir>/preview.parquet` (preview only) |
+
+## Quick start
+
+Run from the `medical-AI-skills` repo root. Set your key first — detection runs
+on build.nvidia.com:
+
+```bash
+export NVIDIA_API_KEY="nvapi-..."
+
+# Preview 5 rows (cheap; writes the trace parquet)
+python skills/report-anonymization/scripts/anonymize_reports.py \
+  skills/report-anonymization/fixtures/batch00_reports_w_PHI.csv \
+  --output-dir runs/report_anonymization_preview \
+  --num-records 5
+
+# Full run on every row
+python skills/report-anonymization/scripts/anonymize_reports.py \
+  /path/to/reports_w_PHI.csv \
+  --output-dir runs/report_anonymization_full \
+  --full
+```
+
+### Test dataset
+
+A 100-case **synthetic** dataset (generated, contains **no real PHI**) is bundled at
+`data/synthetic_reports_100_w_PHI.csv` (`study_uid`, `report_w_PHI`) for end-to-end
+testing and reproducing `BENCHMARK.md`:
+
+```bash
+python skills/report-anonymization/scripts/anonymize_reports.py \
+  skills/report-anonymization/data/synthetic_reports_100_w_PHI.csv \
+  --output-dir runs/report_anonymization_100 --full
+```
+
+### Key arguments
+
+`REPORTS_CSV --output-dir DIR [--full] [--num-records N] [--text-column report_w_PHI] [--id-column study_uid] [--gliner-threshold 0.3] [--evaluate] [--no-emit-telemetry]`
+
+| Environment variable | Required | Purpose |
+|---|---|---|
+| `NVIDIA_API_KEY` | yes | Auth for the bundled build.nvidia.com providers (GLiNER-PII + LLM). |
+| `NEMO_TELEMETRY_ENABLED` | no | `false` disables NeMo's anonymous run telemetry (same as `--no-emit-telemetry`). |
+
+## Output summary fields
+
+`n_reports`, `records_passed` / `records_failed`, `pass_rate`, `entities`
+(total + per-label counts), `residual_phi_leak` (informational verbatim
+replacement-map check), `pipeline_stages` (per NeMo detection iteration),
+`telemetry` (wall-clock timings, throughput, tiktoken input-token estimate),
+and `artifacts` (written paths).
+
+## Repository layout
+
+```
+report-anonymization/
+├── SKILL.md                     # agent-facing skill definition
+├── README.md                    # this file
+├── skill_manifest.yaml          # I/O contract, runtime, sanity checks
+├── requirements.txt             # nemo-anonymizer + pandas + pyarrow + tiktoken
+├── scripts/
+│   └── anonymize_reports.py     # entrypoint (real NeMo Anonymizer wrapper)
+├── validators/
+│   └── output_schema.json       # output JSON schema
+├── data/
+│   └── synthetic_reports_100_w_PHI.csv  # 100-case synthetic test set (no real PHI)
+├── fixtures/
+│   └── batch00_reports_w_PHI.csv  # small synthetic-PHI preview sample (input)
+├── references/
+│   ├── upstream-nemo-anonymizer.md    # upstream tool reference
+│   └── preview-trace-columns.md       # preview.parquet trace schema
+├── evals/
+│   └── evals.json               # NV-ACES eval dataset
+└── BENCHMARK.md                 # with-skill / without-skill results
+```
+
+## Verification
+
+- Output JSON is gated by `validators/output_schema.json` and the manifest
+  `validation.sanity_checks`.
+- With-vs-without evidence lives in
+  [`docs/anonymization-with-vs-without-experiment.md`](../../docs/anonymization-with-vs-without-experiment.md).
+
+## Limitations
+
+- Detection runs on remote LLMs at build.nvidia.com; requires `NVIDIA_API_KEY`
+  and network access, and output is non-deterministic.
+- Strict GLiNER label mode: only listed PHI types are detected.
+- `residual_phi_leak` is an informational verbatim check, not a completeness
+  guarantee. Not a regulatory de-identifier.
+
+## License
+
+Apache-2.0. See the SPDX headers in the source files.
diff --git a/skills/report-anonymization/SKILL.md b/skills/report-anonymization/SKILL.md
@@ -0,0 +1,107 @@
+---
+name: report-anonymization
+description: "Used for de-identifying English radiology reports with NeMo Anonymizer (GLiNER-PII + LLM): replaces detected PHI with bracketed role tokens like [PATIENT], [DOCTOR], [DATE] (MR-RATE reports stage 01). Not for regulatory de-identification or clinical use."
+license: Apache-2.0
+allowed-tools: Bash
+metadata:
+  author: "NVIDIA MedTech <noreply@nvidia.com>"
+  tags:
+    - MedTech
+    - reports
+    - anonymization
+    - de-identification
+    - PHI
+---
+
+# Report Anonymization
+
+## Purpose
+- Used for de-identifying English radiology reports: PHI entities detected by [NeMo Anonymizer](https://github.com/NVIDIA-NeMo/Anonymizer) (GLiNER-PII detection + LLM augmentation/validation) are replaced in place with bracketed role tokens, e.g. `Patient: John A. Doe` -> `Patient: [PATIENT]`, `MRN: 12345678` -> `MRN: [PATIENT_MRN]`, `Dr. Emily Patel` -> `Dr. [DOCTOR]` (MR-RATE reports_preprocessing stage 01).
+- Not for regulatory de-identification, clinical deployment, autonomous diagnosis, or patient-facing use.
+- Manifest I/O: input is `reports_csv` (CSV with `study_uid` + `report_w_PHI` columns); output is `anonymization_summary` (JSON on stdout) plus `anonymized_reports.csv` and `run_report.json` in the output directory.
+
+## Instructions
+- Read `skill_manifest.yaml` before changing arguments, side effects, or validation gates.
+- Set `NVIDIA_API_KEY` first: detection runs on remote models at build.nvidia.com. Without it the run fails with a connection/auth error.
+- Run `scripts/anonymize_reports.py` through the documented command below. Pass the input CSV as the positional argument and a caller-provided run directory as `--output-dir`. Use `--full` to process every row; omit it to preview `--num-records` rows (cheap iteration that also writes `preview.parquet`).
+- If a host agent exposes `run_script`, use `run_script("scripts/anonymize_reports.py", args=[REPORTS_CSV, "--output-dir", OUT, "--full"])`; otherwise run the Bash/Python command shown below.
+- Do not hand-write a NeMo Anonymizer invocation for normal runs; this wrapper owns config, output shaping, telemetry, and the residual-leak audit. See `references/upstream-nemo-anonymizer.md` for the upstream contract.
+- The single JSON object on stdout is the machine-readable summary; progress logs go to stderr.
+
+## Available Scripts
+| Script | Purpose | Arguments |
+|---|---|---|
+| `scripts/anonymize_reports.py` | Primary entrypoint declared by `skill_manifest.yaml`. | `REPORTS_CSV --output-dir DIR [--full] [--num-records N] [--text-column report_w_PHI] [--id-column study_uid] [--gliner-threshold 0.3] [--evaluate] [--no-emit-telemetry]` |
+
+## Prerequisites
+- Python 3.11+ with `nemo-anonymizer>=0.2.1`, `pandas`, `pyarrow`, and `tiktoken` (see `requirements.txt`).
+- `NVIDIA_API_KEY` for the bundled build.nvidia.com providers (GLiNER-PII detector + `openai/gpt-oss-120b` validator/augmenter).
+- Network access to `https://integrate.api.nvidia.com`.
+
+| Variable | Required | Purpose |
+|---|---|---|
+| `NVIDIA_API_KEY` | yes | Auth for the bundled build.nvidia.com model providers used for detection. |
+| `NEMO_TELEMETRY_ENABLED` | no | Set to `false` to disable NeMo Anonymizer's anonymous run telemetry (or pass `--no-emit-telemetry`). |
+
+## Usage
+
+Preview a few rows (cheap; writes `preview.parquet` trace):
+
+```bash
+export NVIDIA_API_KEY="nvapi-..."
+python skills/report-anonymization/scripts/anonymize_reports.py \
+  skills/report-anonymization/fixtures/batch00_reports_w_PHI.csv \
+  --output-dir runs/report_anonymization_preview \
+  --num-records 5
+```
+
+Full run on every row:
+
+```bash
+export NVIDIA_API_KEY="nvapi-..."
+python skills/report-anonymization/scripts/anonymize_reports.py \
+  /path/to/reports_w_PHI.csv \
+  --output-dir runs/report_anonymization_full \
+  --full
+```
+
+### Bundled test dataset
+
+A 100-case **synthetic** test dataset (generated, no real PHI) is included at
+`data/synthetic_reports_100_w_PHI.csv` (columns `study_uid`, `report_w_PHI`) so you
+can exercise the skill end-to-end and reproduce the `BENCHMARK.md` results:
+
+```bash
+export NVIDIA_API_KEY="nvapi-..."
+python skills/report-anonymization/scripts/anonymize_reports.py \
+  skills/report-anonymization/data/synthetic_reports_100_w_PHI.csv \
+  --output-dir runs/report_anonymization_100 \
+  --full
+```
+
+The smaller `fixtures/batch00_reports_w_PHI.csv` remains the quick preview sample.
+
+Evidence pack via the eval engine:
+
+```bash
+python -m eval_engine.run skills/report-anonymization \
+  --fixture skills/report-anonymization/fixtures/batch00_reports_w_PHI.csv \
+  --out runs/report_anonymization_pack
+```
+
+The stdout `anonymization_summary` JSON includes `n_reports`, `records_passed`/`records_failed`, `entities` (total + per-label counts), `residual_phi_leak` (informational verbatim replacement-map check), `pipeline_stages` (per NeMo detection iteration), and `telemetry` (wall-clock timings, throughput, and a tiktoken input-token estimate). The anonymized text is written to `<output-dir>/anonymized_reports.csv` (columns `study_uid,report`) and the full report to `<output-dir>/run_report.json`.
+
+## Limitations
+- Detection/validation run on remote LLMs at build.nvidia.com; the skill requires `NVIDIA_API_KEY` and network access and its output is non-deterministic (hence `reproducibility.mode: preflight`).
+- Strict GLiNER label mode: only the listed PHI entity types are detected; entity types the detector never proposes are not replaced.
+- `residual_phi_leak` is an informational verbatim replacement-map check, not a completeness guarantee; it does not detect PHI the model never mapped and skips originals shorter than 3 characters.
+- Redact strategy only. Upstream NeMo Anonymizer also offers Substitute, Annotate, Hash, and Rewrite; those are out of scope for this stage-01 wrapper.
+- Not for clinical deployment, regulatory de-identification, autonomous diagnosis, or patient-facing use.
+
+## Troubleshooting
+| Error | Cause | Fix |
+|---|---|---|
+| `Text column 'report_w_PHI' not in ...` | Wrong column or delimiter. | Pass `--text-column`/`--id-column`, or fix the CSV header. |
+| `Workflow failed: Connection to model ... failed while running health checks` | `NVIDIA_API_KEY` unset/invalid, or no network to build.nvidia.com. | Export a valid `NVIDIA_API_KEY` and confirm access to `https://integrate.api.nvidia.com`. |
+| Non-zero exit with `records_failed > 0` | Records dropped mid-pipeline (rate limits, transient API errors). | Inspect `run_report.json` `failed_records`; re-run. Dropped rows are infra issues, not strategy issues. |
+| `No module named 'anonymizer'` | `nemo-anonymizer` not installed in the active env. | `pip install -r skills/report-anonymization/requirements.txt`. |