From fcfec9dafa1e030f7ac2b65de1c8db81d7b21558 Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Sat, 21 Mar 2026 14:08:06 +0000 Subject: [PATCH 01/22] Initial plan From 95f230cc4c675284a50e89442636a1a43725cc57 Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Sat, 21 Mar 2026 14:10:10 +0000 Subject: [PATCH 02/22] Add ADR-001: Simplified Evaluation Pipeline Using DataFrame as Clean Interface Co-authored-by: omri374 <3776619+omri374@users.noreply.github.com> Agent-Logs-Url: https://github.com/microsoft/presidio-research/sessions/eb50120c-0b18-41ca-9db2-55c071434576 --- .../ADR-001-simplified-evaluation-pipeline.md | 166 ++++++++++++++++++ 1 file changed, 166 insertions(+) create mode 100644 docs/adr/ADR-001-simplified-evaluation-pipeline.md diff --git a/docs/adr/ADR-001-simplified-evaluation-pipeline.md b/docs/adr/ADR-001-simplified-evaluation-pipeline.md new file mode 100644 index 00000000..6164217a --- /dev/null +++ b/docs/adr/ADR-001-simplified-evaluation-pipeline.md @@ -0,0 +1,166 @@ +# ADR-001: Simplified Evaluation Pipeline Using DataFrame as Clean Interface + +## Status + +Proposed + +## Date + +2026-03-21 + +## Context + +The current evaluation pipeline in `presidio_evaluator` is tightly coupled and requires several intermediate steps to go from a dataset and a model to evaluation metrics. The full flow today is: + +``` +Dataset (List[InputSample]) + → Evaluator(model=model) # model is embedded in evaluator + → evaluator.evaluate_all(dataset) # runs inference AND builds per-sample EvaluationResult objects + → List[EvaluationResult] # intermediate per-sample carriers + → evaluator.calculate_score(...) # calls get_results_dataframe() → DataFrame → scoring + → EvaluationResult (aggregated) +``` + +This design has four concrete pain points: + +1. **Model is coupled to the Evaluator** — `BaseEvaluator.__init__` takes a `model` argument, and `evaluate_all()` calls `model.batch_predict`, `model.filter_tags_in_supported_entities`, and `model.to_scheme` internally. This makes it impossible to evaluate a pre-computed result set without also instantiating a model. + +2. **`evaluate_all()` does two things** — it runs model inference AND builds per-sample `EvaluationResult` objects. These objects are simple data carriers holding `(tokens, actual_tags, predicted_tags, start_indices)`, yet they require callers to go through the evaluator just to get predictions into a usable shape. + +3. **The real interface already exists but is buried** — `SpanEvaluator.calculate_score()` and `TokenEvaluator.calculate_score()` internally call `get_results_dataframe()` to convert `List[EvaluationResult]` to a DataFrame, and then call `calculate_score_on_df()`. The `EvaluationResult` list is a wasteful intermediate step; the DataFrame is the actual computational surface. + +4. **Entity mapping logic is scattered** — entity type remapping lives in `BaseModel.align_entity_types`, `BaseEvaluator.align_entity_types` (static method), and is applied independently in both models and evaluators. There is no single, composable entry point for this transformation. + +A new contributor trying to run an evaluation must understand all of these layers before being able to get a single metric out of the system. The goal of this ADR is to establish a clean, linear pipeline with a single well-defined interface point between the model and the evaluator. + +## Decision + +Adopt the **results DataFrame** as the canonical interface boundary between the model stage and the evaluation stage. The DataFrame has the following five-column schema: + +| Column | Type | Description | +|---|---|---| +| `sentence_id` | `int` | Index of the source sentence in the dataset | +| `token` | `str` | Token string | +| `annotation` | `str` | Ground-truth entity tag (from `InputSample.tags`) | +| `prediction` | `str` | Model-predicted entity tag | +| `start_indices` | `int` | Character start offset of the token within the sentence | + +This schema is already produced by `get_results_dataframe()` and consumed by `calculate_score_on_df()`. The changes below make it the first-class output of the model stage instead of an internal implementation detail of the evaluator. + +### Proposed simplified pipeline (5 steps) + +```python +# 1. Load dataset +dataset = InputSample.read_dataset_json("data/dataset.json") + +# 2. Choose model and run predictions → get DataFrame directly +model = SpacyModel(model=nlp) # or PresidioAnalyzerWrapper, etc. +results_df = model.predict_dataset(dataset) # NEW: returns the DataFrame directly + +# 3. Map entities (optional, operates on DataFrame) +results_df = map_entities(results_df, mapping={"GPE": "LOCATION", "LOC": "LOCATION"}) + +# 4. Score (evaluator is model-free, takes only the DataFrame) +evaluator = SpanEvaluator() # no model argument needed! +results = evaluator.calculate_score_on_df(per_type=True, results_df=results_df) + +# 5. Analyze/plot +plotter = Plotter(results=results) +plotter.plot_scores() +``` + +### Concrete changes + +| Component | Current | Proposed | +|---|---|---| +| `BaseModel` | `predict()` → `List[str]` tags per sample | Add `predict_dataset()` → `pd.DataFrame` with the 5-column schema | +| `BaseEvaluator` | `__init__(model=...)`, `evaluate_all(dataset)` | `model` becomes optional; primary entry point becomes `calculate_score_on_df()` | +| Entity mapping | Scattered across `BaseModel`, `BaseEvaluator`, and `InputSample` | Single `map_entities(df, mapping)` utility operating on the DataFrame | +| `evaluate_all()` | Required step in the pipeline | Becomes a convenience wrapper (calls `predict_dataset` then `calculate_score_on_df`) | +| `EvaluationResult` (per-sample) | Carrier object for tokens/tags | Eliminated — the DataFrame IS the carrier | + +### `predict_dataset` method on `BaseModel` + +```python +def predict_dataset(self, dataset: List[InputSample], **kwargs) -> pd.DataFrame: + """Run model on dataset and return a standardized results DataFrame.""" + predictions = self.batch_predict(dataset, **kwargs) + rows = [] + for i, (sample, prediction) in enumerate(zip(dataset, predictions)): + prediction = self.filter_tags_in_supported_entities(prediction) + prediction = self.to_scheme(prediction) + for token, tag, pred, start in zip( + sample.tokens, sample.tags, prediction, sample.start_indices + ): + rows.append({ + "sentence_id": i, + "token": str(token), + "annotation": tag, + "prediction": pred, + "start_indices": start, + }) + return pd.DataFrame(rows) +``` + +### `map_entities` utility function + +```python +def map_entities(df: pd.DataFrame, mapping: Dict[str, str]) -> pd.DataFrame: + """Map entity types in both annotation and prediction columns.""" + df = df.copy() + df["annotation"] = df["annotation"].replace(mapping) + df["prediction"] = df["prediction"].replace(mapping) + return df +``` + +## Consequences + +### Positive + +- **Reduced conceptual overhead** — new contributors only need to understand `predict_dataset` + `calculate_score_on_df`; the full evaluator internals are optional knowledge. +- **Decoupled model and evaluator** — models and evaluators can be developed, tested, and swapped independently. `SpanEvaluator` and `TokenEvaluator` no longer require a `model` instance. +- **Composable entity mapping** — `map_entities` is a pure function on DataFrames; it can be chained, applied selectively, or omitted without touching model or evaluator configuration. +- **Easier offline/batch evaluation** — a results DataFrame can be saved to disk and re-evaluated later without re-running inference. This is currently impossible without serializing `List[EvaluationResult]` objects. +- **`calculate_score_on_df` already exists and is tested** — `SpanEvaluator.calculate_score_on_df` and `TokenEvaluator.calculate_score_on_df` are already implemented and covered by tests in `test_span_evaluator.py`. The new pipeline plugs directly into existing code. +- **Eliminates the `EvaluationResult` per-sample carrier** — removes a class whose only purpose is to ferry data between `evaluate_all()` and `get_results_dataframe()`. + +### Negative / Trade-offs + +- **`predict_dataset` materializes all predictions in memory** — for very large datasets, the full DataFrame is held in RAM. The current `evaluate_all()` loop can in principle be streamed (though it isn't today). +- **`evaluate_all()` becomes a thin wrapper** — code that currently calls `evaluate_all()` and inspects `List[EvaluationResult]` directly will need to be updated if it relies on the per-sample `EvaluationResult` structure. Backward compatibility is preserved at the call site, but the internal representation changes. +- **Schema enforcement is implicit** — the 5-column schema is a convention, not enforced by a type. Callers that hand-construct DataFrames must respect column names and types. +- **`EvaluationResult` (per-sample) removal is a breaking change** — any downstream code (e.g., notebooks, scripts) that imports or type-annotates `EvaluationResult` will need updating. A deprecation period should be provided. + +## Implementation Plan + +1. **Add `BaseModel.predict_dataset()`** — implement the method as sketched above in `presidio_evaluator/models/base_model.py`. Add a unit test in `tests/` that verifies the 5-column schema and correct row count for a small synthetic dataset. + +2. **Add `map_entities()` utility** — add the function (and `Dict` import) to `presidio_evaluator/evaluation/` (e.g., in a new `utils.py` or alongside `get_results_dataframe`). Add a unit test verifying that both `annotation` and `prediction` columns are remapped. + +3. **Make `model` optional in `BaseEvaluator`** — change `BaseEvaluator.__init__(self, model, ...)` so that `model` defaults to `None`. Add a runtime check that raises a clear error if `evaluate_all()` is called when `model is None`. + +4. **Update `evaluate_all()` to delegate to `predict_dataset` + `calculate_score_on_df`** — refactor `SpanEvaluator.evaluate_all()` and `TokenEvaluator.evaluate_all()` to call `self.model.predict_dataset(dataset)` and then pass the result to `calculate_score_on_df()`. This ensures a single code path for both old and new usage. + +5. **Deprecate per-sample `EvaluationResult`** — add a `DeprecationWarning` to the `EvaluationResult` class (the per-sample variant) and update `get_results_dataframe()` to note it will be removed in a future release. + +6. **Update documentation and notebooks** — revise `docs/evaluation.md` and any Jupyter notebooks under `notebooks/` to demonstrate the new 5-step pipeline. Keep the old `evaluate_all()` example with a note that it is the backward-compatible convenience path. + +7. **Add integration test** — add an end-to-end test that runs `predict_dataset` → `map_entities` → `calculate_score_on_df` on a small synthetic dataset and asserts that the returned metrics match expected values. + +## Alternatives Considered + +### Keep the current architecture and add a thin adapter function + +A helper function like `run_evaluation(model, dataset, evaluator)` could hide the complexity without changing any classes. This was rejected because it does not address the root coupling: `evaluate_all()` would still conflate inference with result formatting, and entity mapping would remain scattered. + +### Use a custom result object instead of a DataFrame + +A typed dataclass or `NamedTuple` (e.g., `TokenPrediction`) could replace the per-sample `EvaluationResult`. This was rejected because `calculate_score_on_df` already exists and is the correct computational surface. Introducing another intermediate type would add a conversion step rather than remove one. + +### Make `BaseModel` return a list of tagged spans instead of a DataFrame + +Returning `List[List[Tuple[str, str, str, int]]]` (one list per sentence) would preserve laziness but lose the benefits of columnar operations and Pandas compatibility. Rejected because the DataFrame is already the internal format used for all scoring, and forcing a conversion at the last moment adds no value. + +### Move entity mapping into `BaseModel.predict_dataset` as a parameter + +Passing `mapping` directly to `predict_dataset(dataset, entity_mapping=...)` would keep mapping close to prediction. This was rejected in favor of a standalone `map_entities` utility because it makes the transformation composable and testable in isolation, and because mapping is a property of the evaluation goal (what entities to score), not of the model. From b59cdedabd37ca392630e1b7dd2cd3a26e825953 Mon Sep 17 00:00:00 2001 From: Omri Mendels Date: Sat, 21 Mar 2026 16:19:59 +0200 Subject: [PATCH 03/22] Simplify evaluation pipeline and update model prediction Updated the evaluation pipeline to simplify the model prediction process and entity mapping. Removed redundant code and clarified comments for better understanding. --- .../ADR-001-simplified-evaluation-pipeline.md | 41 ++----------------- 1 file changed, 4 insertions(+), 37 deletions(-) diff --git a/docs/adr/ADR-001-simplified-evaluation-pipeline.md b/docs/adr/ADR-001-simplified-evaluation-pipeline.md index 6164217a..6ab8fc8d 100644 --- a/docs/adr/ADR-001-simplified-evaluation-pipeline.md +++ b/docs/adr/ADR-001-simplified-evaluation-pipeline.md @@ -54,13 +54,13 @@ This schema is already produced by `get_results_dataframe()` and consumed by `ca dataset = InputSample.read_dataset_json("data/dataset.json") # 2. Choose model and run predictions → get DataFrame directly -model = SpacyModel(model=nlp) # or PresidioAnalyzerWrapper, etc. +model = PresidioAnalyzerWrapper(model=AnalyzerEngine()) results_df = model.predict_dataset(dataset) # NEW: returns the DataFrame directly -# 3. Map entities (optional, operates on DataFrame) -results_df = map_entities(results_df, mapping={"GPE": "LOCATION", "LOC": "LOCATION"}) +# 3. Map entities (transforms both predictions and annotations into canonical entities) +results_df = map_entities(results_df) -# 4. Score (evaluator is model-free, takes only the DataFrame) +# 4. Score evaluator = SpanEvaluator() # no model argument needed! results = evaluator.calculate_score_on_df(per_type=True, results_df=results_df) @@ -79,39 +79,6 @@ plotter.plot_scores() | `evaluate_all()` | Required step in the pipeline | Becomes a convenience wrapper (calls `predict_dataset` then `calculate_score_on_df`) | | `EvaluationResult` (per-sample) | Carrier object for tokens/tags | Eliminated — the DataFrame IS the carrier | -### `predict_dataset` method on `BaseModel` - -```python -def predict_dataset(self, dataset: List[InputSample], **kwargs) -> pd.DataFrame: - """Run model on dataset and return a standardized results DataFrame.""" - predictions = self.batch_predict(dataset, **kwargs) - rows = [] - for i, (sample, prediction) in enumerate(zip(dataset, predictions)): - prediction = self.filter_tags_in_supported_entities(prediction) - prediction = self.to_scheme(prediction) - for token, tag, pred, start in zip( - sample.tokens, sample.tags, prediction, sample.start_indices - ): - rows.append({ - "sentence_id": i, - "token": str(token), - "annotation": tag, - "prediction": pred, - "start_indices": start, - }) - return pd.DataFrame(rows) -``` - -### `map_entities` utility function - -```python -def map_entities(df: pd.DataFrame, mapping: Dict[str, str]) -> pd.DataFrame: - """Map entity types in both annotation and prediction columns.""" - df = df.copy() - df["annotation"] = df["annotation"].replace(mapping) - df["prediction"] = df["prediction"].replace(mapping) - return df -``` ## Consequences From 1df517c6353c5fdcf4e855a5337e215b610050b7 Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Sat, 21 Mar 2026 14:27:48 +0000 Subject: [PATCH 04/22] docs: add ADR-002 for entity mapping via CanonicalMapper Co-authored-by: omri374 <3776619+omri374@users.noreply.github.com> Agent-Logs-Url: https://github.com/microsoft/presidio-research/sessions/f2aad060-8eac-4214-aba8-01de85cf7536 --- docs/adr/ADR-002-entity-mapping.md | 172 +++++++++++++++++++++++++++++ 1 file changed, 172 insertions(+) create mode 100644 docs/adr/ADR-002-entity-mapping.md diff --git a/docs/adr/ADR-002-entity-mapping.md b/docs/adr/ADR-002-entity-mapping.md new file mode 100644 index 00000000..826b24b1 --- /dev/null +++ b/docs/adr/ADR-002-entity-mapping.md @@ -0,0 +1,172 @@ +# ADR-002: Entity Mapping via CanonicalMapper + +## Status + +Proposed + +## Date + +2026-03-21 + +## Context + +Users of Presidio Evaluator bring their own entity label vocabularies — from custom datasets, +fine-tuned models, or third-party tools. Before evaluation can happen, every user-defined label +must be resolved to a canonical entity so that dataset annotations and model predictions can be +compared on equal footing. + +The current approach has five pain points: + +1. **Mapping logic is scattered** — entity type remapping lives in `BaseModel.align_entity_types`, + `BaseEvaluator.align_entity_types`, and `InputSample` processing. There is no single, reusable + entry point for the transformation. + +2. **No structured resolution pipeline** — the existing code does not distinguish between labels + that are exact matches, fuzzy matches, or unknown. All unmapped labels are silently dropped or + cause errors, giving users no visibility into what happened. + +3. **Semantic-similarity dependency** — the current `SemanticEntityMapper` relies on + `sentence-transformers` to find approximate matches. This adds a heavy ML dependency for a task + that can be solved adequately with lightweight string matching against a known vocabulary. + +4. **No interactive fallback** — when a label cannot be resolved automatically, there is no guided + way for the user to supply the correct canonical entity. They must edit source code or + configuration files directly. + +5. **No audit trail** — users cannot tell how each label was resolved (exact match, approximate + match, country-prefix logic, manual entry) without reading the source code. + +## Decision + +Introduce a single, stateful class — `CanonicalMapper` — as the sole entry point for resolving +user-defined labels to canonical entities. The same class is used for both sides of an evaluation: +the dataset's label vocabulary and the model's output label vocabulary. Both are resolved +independently; evaluation then compares canonical-to-canonical. + +### Resolution tiers + +`CanonicalMapper` attempts each tier in order, stopping at the first match: + +| Resolution Tier | Matching Condition | Output | +|---|---|---| +| **EXACT** | Label (after normalisation) matches a known alias | Canonical entity | +| **COUNTRY** | Label begins with a recognised country prefix; remainder resolves to a known document type | Canonical entity | +| **COUNTRY_FALLBACK** | Label begins with a recognised country prefix; remainder is not a known document type | `NATIONAL_ID` (with warning) | +| **FUZZY** | Approximate string match against the alias vocabulary meets the confidence threshold | Canonical entity | +| **PENDING** | None of the above | Requires user action | + +After the user resolves pending labels (via `map()` or `resolve_interactively()`), resolved labels +are tagged **MANUAL** or **NONE** (suppressed from evaluation). + +Before any tier is attempted, BIO/BIOES/BILOU tagging scheme prefixes and suffixes are stripped +transparently (e.g. `B-PERSON` is looked up as `PERSON`). The original label remains the key in +all outputs. + +### Workflow + +``` +CanonicalMapper(labels) + → auto-resolve pass (EXACT → COUNTRY → COUNTRY_FALLBACK → FUZZY) + → inspect .pending # labels that need attention + → .map({...}) # programmatic assignment + → .resolve_interactively() # guided terminal/notebook prompts + → .get_mapping() # returns dict[str, str | None] +``` + +When all labels auto-resolve, `.get_mapping()` can be called immediately — no additional steps +are needed. When pending labels remain, `.get_mapping()` raises `IncompleteMapping` until every +label is accounted for. + +### Typical usage + +```python +# `from_dataset` is a convenience constructor: it extracts entity labels from the +# dataset's spans, runs the auto-resolve pass, and returns the mapping dict directly +# if every label resolves automatically. When labels are still pending it returns +# the CanonicalMapper instance so the caller can resolve them before calling +# .get_mapping(). + +# Everything auto-resolves — mapping dict returned immediately +mapping = CanonicalMapper.from_dataset(samples) + +# Some labels are pending — resolve interactively, then retrieve +mapper = CanonicalMapper(["PERSON", "EMAIL_ADDRESS", "MY_CUSTOM_LABEL"]) +mapper.resolve_interactively() +mapping = mapper.get_mapping() + +# Programmatic (batch) assignment — no prompts +mapper.map({"MY_CUSTOM_LABEL": "PERSON", "INTERNAL_ID": None}) +mapping = mapper.get_mapping() +``` + +### Logging + +Every resolution decision is logged at `INFO` level, with country-prefix fallback at `WARNING` +and unresolvable labels at `WARNING`. A summary line after the auto-resolve pass reports how many +labels were resolved, how many were fuzzy, and how many are pending. This gives users a full audit +trail without inspecting internal state. + +### Removing the semantic-similarity dependency + +`sentence-transformers` is removed from the project dependencies entirely. All entity resolution +goes through the `EntityHierarchy` alias vocabulary and lightweight string matching — no +embeddings, no external ML models. + +## Consequences + +### Positive + +- **Single resolution entry point** — all mapping logic lives in one place; scattered + `align_entity_types` calls across `BaseModel` and `BaseEvaluator` are replaced. +- **Transparency** — every resolution decision is logged and visible in the HTML audit table + (`render_html()`). Users always know how each label was handled. +- **Lighter dependency footprint** — removing `sentence-transformers` eliminates a large + transitive dependency chain (PyTorch, transformers, tokenizers) from the default install. +- **Guided manual resolution** — `resolve_interactively()` shows ranked fuzzy suggestions for + pending labels, making manual mapping fast even for large vocabularies. +- **Composable** — `get_mapping()` returns a plain `dict[str, str | None]` that can be passed + to any downstream evaluation code, serialised, or version-controlled without additional tooling. +- **Atomic batch updates** — `map()` validates all entries before applying any, preventing + partial state corruption. + +### Negative / Trade-offs + +- **Breaking change for `SemanticEntityMapper` users** — the class is removed; callers must + migrate to `CanonicalMapper`. No automatic migration path is provided. +- **Resolution quality ceiling** — fuzzy string matching has lower recall on abbreviated or + domain-specific labels than semantic similarity. Users with highly non-standard vocabularies + may have more labels land in `pending`. +- **Stateful object** — `CanonicalMapper` holds mutable resolution state. Callers that share an + instance across threads or processes must handle concurrency themselves. +- **`IncompleteMapping` is a hard stop** — pipelines that previously silently dropped unknown + labels will now fail explicitly until all labels are resolved or suppressed. This is intentional + but requires pipeline changes for existing users. + +## Alternatives Considered + +### Keep `SemanticEntityMapper` and patch it + +Extending the existing semantic-similarity approach with better logging and an interactive +fallback was considered. This was rejected because the `sentence-transformers` dependency is +disproportionate to the task: resolving a finite, known vocabulary of entity labels does not +benefit from embeddings, and the dependency imposes significant install-time and runtime costs. + +### A stateless mapping function + +A pure function `resolve_labels(labels, hierarchy) → dict` with no interactive capability was +considered. This was rejected because it cannot handle the interactive fallback path needed when +labels cannot be auto-resolved. Callers would have to implement their own retry loops, defeating +the goal of a single reusable entry point. + +### Embed mapping inside `BaseModel.predict_dataset` + +Passing an entity mapping directly to `predict_dataset` was considered (see ADR-001). This was +rejected because mapping is a property of the *evaluation goal* (which canonical entities to +score), not of the model. Keeping it separate makes both components independently testable and +allows the same model output to be evaluated under different mapping configurations. + +### A fully automatic mapping with no `pending` state + +Resolving every unknown label to a default value (e.g. `None`) without surfacing failures was +considered. This was rejected because silent data loss is worse than an explicit error. Surfacing +`pending` labels ensures users make conscious decisions about every label in their vocabulary. From 3e900aafb9cd0203c794816cd43c61808d2bd3bf Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Sat, 21 Mar 2026 14:52:17 +0000 Subject: [PATCH 05/22] docs: revise ADR-002 per review feedback Co-authored-by: omri374 <3776619+omri374@users.noreply.github.com> Agent-Logs-Url: https://github.com/microsoft/presidio-research/sessions/7e819255-deb4-44b4-88fe-6ef94a3afcdc --- docs/adr/ADR-002-entity-mapping.md | 94 +++++++++++++----------------- 1 file changed, 42 insertions(+), 52 deletions(-) diff --git a/docs/adr/ADR-002-entity-mapping.md b/docs/adr/ADR-002-entity-mapping.md index 826b24b1..2c140acc 100644 --- a/docs/adr/ADR-002-entity-mapping.md +++ b/docs/adr/ADR-002-entity-mapping.md @@ -15,33 +15,28 @@ fine-tuned models, or third-party tools. Before evaluation can happen, every use must be resolved to a canonical entity so that dataset annotations and model predictions can be compared on equal footing. -The current approach has five pain points: +The current approach has two pain points: -1. **Mapping logic is scattered** — entity type remapping lives in `BaseModel.align_entity_types`, - `BaseEvaluator.align_entity_types`, and `InputSample` processing. There is no single, reusable - entry point for the transformation. +1. **Mapping is required for meaningful cross-model comparison, but the current approach is too simplistic** — comparing different models against a shared dataset requires that every model's predictions and the dataset's annotations use a common label vocabulary. Without reliable mapping, results are biased: labels that should be considered equivalent are treated as different, inflating false-negative counts and depressing recall. The existing mapping code handles only the simplest cases; real-world label vocabularies include tagging-scheme prefixes, country-prefixed document types, and near-synonyms that the current logic silently drops or fails on, requiring significant manual intervention to get trustworthy numbers. 2. **No structured resolution pipeline** — the existing code does not distinguish between labels that are exact matches, fuzzy matches, or unknown. All unmapped labels are silently dropped or cause errors, giving users no visibility into what happened. -3. **Semantic-similarity dependency** — the current `SemanticEntityMapper` relies on - `sentence-transformers` to find approximate matches. This adds a heavy ML dependency for a task - that can be solved adequately with lightweight string matching against a known vocabulary. - -4. **No interactive fallback** — when a label cannot be resolved automatically, there is no guided - way for the user to supply the correct canonical entity. They must edit source code or - configuration files directly. - -5. **No audit trail** — users cannot tell how each label was resolved (exact match, approximate - match, country-prefix logic, manual entry) without reading the source code. - ## Decision Introduce a single, stateful class — `CanonicalMapper` — as the sole entry point for resolving -user-defined labels to canonical entities. The same class is used for both sides of an evaluation: -the dataset's label vocabulary and the model's output label vocabulary. Both are resolved -independently; evaluation then compares canonical-to-canonical. +user-defined labels to **canonical entities**. A canonical entity is a normalised, taxonomy-defined +label drawn from the `EntityHierarchy` vocabulary (e.g. `PERSON`, `EMAIL_ADDRESS`, +`PASSPORT`). Mapping a raw label to a canonical entity means finding the single taxonomy entry that +best represents the concept the raw label describes. Once every label on both sides of an evaluation +— dataset annotations and model predictions — has been mapped to a canonical entity, scores can be +computed fairly: identical canonical entities count as matches regardless of how the two sides +originally spelled or tagged them. + +The same class is used for both sides of an evaluation: the dataset's label vocabulary and the +model's output label vocabulary. Both are resolved independently; evaluation then compares +canonical-to-canonical. ### Resolution tiers @@ -106,12 +101,6 @@ and unresolvable labels at `WARNING`. A summary line after the auto-resolve pass labels were resolved, how many were fuzzy, and how many are pending. This gives users a full audit trail without inspecting internal state. -### Removing the semantic-similarity dependency - -`sentence-transformers` is removed from the project dependencies entirely. All entity resolution -goes through the `EntityHierarchy` alias vocabulary and lightweight string matching — no -embeddings, no external ML models. - ## Consequences ### Positive @@ -120,8 +109,6 @@ embeddings, no external ML models. `align_entity_types` calls across `BaseModel` and `BaseEvaluator` are replaced. - **Transparency** — every resolution decision is logged and visible in the HTML audit table (`render_html()`). Users always know how each label was handled. -- **Lighter dependency footprint** — removing `sentence-transformers` eliminates a large - transitive dependency chain (PyTorch, transformers, tokenizers) from the default install. - **Guided manual resolution** — `resolve_interactively()` shows ranked fuzzy suggestions for pending labels, making manual mapping fast even for large vocabularies. - **Composable** — `get_mapping()` returns a plain `dict[str, str | None]` that can be passed @@ -131,42 +118,45 @@ embeddings, no external ML models. ### Negative / Trade-offs -- **Breaking change for `SemanticEntityMapper` users** — the class is removed; callers must - migrate to `CanonicalMapper`. No automatic migration path is provided. -- **Resolution quality ceiling** — fuzzy string matching has lower recall on abbreviated or - domain-specific labels than semantic similarity. Users with highly non-standard vocabularies - may have more labels land in `pending`. -- **Stateful object** — `CanonicalMapper` holds mutable resolution state. Callers that share an - instance across threads or processes must handle concurrency themselves. - **`IncompleteMapping` is a hard stop** — pipelines that previously silently dropped unknown labels will now fail explicitly until all labels are resolved or suppressed. This is intentional but requires pipeline changes for existing users. ## Alternatives Considered -### Keep `SemanticEntityMapper` and patch it +### 1. Score against own labels (no mapping) -Extending the existing semantic-similarity approach with better logging and an interactive -fallback was considered. This was rejected because the `sentence-transformers` dependency is -disproportionate to the task: resolving a finite, known vocabulary of entity labels does not -benefit from embeddings, and the dependency imposes significant install-time and runtime costs. +Each model is evaluated only on the entity types it natively supports, with no mapping at all. +This requires nothing from the user but makes cross-model comparison impossible — models are +evaluated on different sets of entities, so their scores are not comparable. High customizability +since each model is unconstrained, but that freedom undermines any meaningful benchmark. -### A stateless mapping function +### 2. Manual mapping only -A pure function `resolve_labels(labels, hierarchy) → dict` with no interactive capability was -considered. This was rejected because it cannot handle the interactive fallback path needed when -labels cannot be auto-resolved. Callers would have to implement their own retry loops, defeating -the goal of a single reusable entry point. +Require users to supply a complete `dict[str, str]` upfront and apply it verbatim, with no +auto-resolution. This is simple to implement and fully predictable, but it is burdensome for +large or evolving label vocabularies where most mappings are straightforward. Users must enumerate +every label manually, including trivially resolvable ones, before evaluation can start. -### Embed mapping inside `BaseModel.predict_dataset` +### 3. Semantic similarity (embedding-based matching) -Passing an entity mapping directly to `predict_dataset` was considered (see ADR-001). This was -rejected because mapping is a property of the *evaluation goal* (which canonical entities to -score), not of the model. Keeping it separate makes both components independently testable and -allows the same model output to be evaluated under different mapping configurations. +Use a sentence-transformer model to embed both the raw label and all canonical entity names, then +pick the nearest neighbour as the resolved canonical. This can handle abbreviated or +domain-specific labels that string matching misses. It was rejected because it adds a heavy ML +dependency (PyTorch, transformers, tokenizers) for a task whose label vocabulary is finite and +known; the quality gain over fuzzy string matching does not justify the install-time and runtime +overhead. It also makes resolution non-deterministic across model versions. + +### 4. A stateless mapping function -### A fully automatic mapping with no `pending` state +A pure function `resolve_labels(labels, hierarchy) → dict` with no state or interactive +capability was considered. This was rejected because it cannot handle the fallback path for labels +that cannot be auto-resolved — callers would have to implement their own retry loops and conflict +resolution, defeating the goal of a single reusable entry point. -Resolving every unknown label to a default value (e.g. `None`) without surfacing failures was -considered. This was rejected because silent data loss is worse than an explicit error. Surfacing -`pending` labels ensures users make conscious decisions about every label in their vocabulary. +### 5. Embed mapping inside the model prediction step + +Passing an entity mapping directly to the model's `predict` call was considered. This was +rejected because mapping is a property of the *evaluation goal* (which canonical entities to +score), not of the model. Keeping mapping separate makes both components independently testable +and allows the same model output to be evaluated under different mapping configurations. From 6ca1cac85887d1866413f161b69b6ae97cd97499 Mon Sep 17 00:00:00 2001 From: Omri Mendels Date: Sat, 21 Mar 2026 17:06:05 +0200 Subject: [PATCH 06/22] Add hierarchical entity mapping dictionary Added a proposed hierarchical entity mapping dictionary with detailed classifications for various entity types and their corresponding attributes. --- docs/adr/ADR-002-entity-mapping.md | 431 +++++++++++++++++++++++++++++ 1 file changed, 431 insertions(+) diff --git a/docs/adr/ADR-002-entity-mapping.md b/docs/adr/ADR-002-entity-mapping.md index 2c140acc..6d201633 100644 --- a/docs/adr/ADR-002-entity-mapping.md +++ b/docs/adr/ADR-002-entity-mapping.md @@ -160,3 +160,434 @@ Passing an entity mapping directly to the model's `predict` call was considered. rejected because mapping is a property of the *evaluation goal* (which canonical entities to score), not of the model. Keeping mapping separate makes both components independently testable and allows the same model output to be evaluated under different mapping configurations. + + + +## Proposed hierarchical entity mapping dictionary +All entities are mapped to the 3rd level (canonoical) +2nd level: PERSON, DEMOGRAPHIC, CONTACT, LOCATION, ORGANIZATION, EMPLOYMENT, GOVERNMENT_ID, FINANCIAL_PII, DEVICE_IDENTIFIER, BIOMETRIC, NETWORK_IDENTIFIER, AUTHENTICATION, PHI, VEHICLE_PII, LEGAL_PII, TRAVEL_PII, EDUCATION, DATE_TIME +3rd level: NAME, ..., TITLE, USERNAME, ..., AGE, GENDER, ..., ADDRESS, ..., COMPANY, +Although the user can choose a different level of granularity (coarser or broader + +```py +HIERARCHY: dict = { + "PII": { + "PERSON": { + "NAME": { + "FIRST_NAME": [ + "FIRSTNAME", + "NAME_GIVEN", + "GIVENNAME", + "GIVENNAME1", + "GIVENNAME2", + ], + "MIDDLE_NAME": ["MIDDLENAME"], + "LAST_NAME": [ + "LASTNAME", + "LASTNAME1", + "LASTNAME2", + "LASTNAME3", + "SURNAME", + "NAME_FAMILY", + ], + "FULL_NAME": [ + "FULLNAME", + "DOCTOR", + "PATIENT_NAME", + "DOCTOR_NAME", + "HCW", + "NAME_MEDICAL_PROFESSIONAL", + ], + "MAIDEN_NAME": [], + "PER": [], + }, + "PREFIX": [], + "SUFFIX": [], + "TITLE": [], + "USERNAME": ["USER_NAME", "DISPLAYNAME", "ACCOUNTNAME", "ACCOUNT_NAME"], + "ALIAS": ["OTHER_NAME"], + }, + "DEMOGRAPHIC": { + "AGE": ["AGE_GROUP", "AGE_RANGE", "AGE_IN_YEARS"], + "GENDER": ["SEX", "SEXTYPE"], + "SEXUAL_ORIENTATION": ["SEXUALITY"], + "RELIGION": ["RELIGIOUS_BELIEF", "BELIEF"], + "ETHNICITY": ["RACE_ETHNICITY", "ORIGIN", "RACE"], + "NATIONALITY": ["NRP", "NORP"], + "MARITAL_STATUS": [], + "LANGUAGE": [], + "POLITICAL_AFFILIATION": ["POLITICAL_VIEW"], + "ZODIAC_SIGN": [], + "DEMOGRAPHIC_ATTRIBUTE": ["DEM"], + "PHYSICAL_DESCRIPTOR": { + "PHYSICAL_ATTRIBUTE": [], + "SKIN_COLOR": ["SKIN_TONE", "COMPLEXION"], + "EYE_COLOR": ["EYECOLOR"], + "HAIR_COLOR": ["HAIRCOLOR"], + "HEIGHT": [], + "WEIGHT": [], + "BODY_MEASUREMENT": ["BODY_MEASURE", "MEASUREMENTS"], + }, + }, + "CONTACT": { + "EMAIL_ADDRESS": ["EMAIL", "EMA"], + "PHONE_NUMBER": ["PHONE", "TEL", "TELEPHONENUM", "PHONENUMBER", "PHN", "MOBILE"], + "FAX": ["FAX_NUMBER"], + "SOCIAL_HANDLE": ["QQ"], # QQ: Chinese messaging platform ID + # URL lives only under NETWORK_IDENTIFIER to avoid ambiguity + "WEBSITE": ["DOMAIN", "WEB"], + }, + "LOCATION": { + "ADDRESS": { + "STREET_ADDRESS": [ + "STREET", + "STREETADDRESS", + "LOCATION_ADDRESS", + "LOCATION_ADDRESS_STREET", + "ADDRESS", + ], + "BUILDING_NUMBER": ["BUILDINGNUMBER", "BUILDINGNUM"], + "SECONDARY_ADDRESS": ["SECONDARYADDRESS", "SECADDRESS"], + "CITY": ["LOCATION_CITY"], + "COUNTY": ["PROVINCE"], + "STATE": ["LOCATION_STATE"], + "POSTAL_CODE": [ + "ZIPCODE", + "ZIP", + "POSTCODE", + "LOCATION_ZIP", + "UK_POSTCODE", + "CEP", # BR Código de Endereçamento Postal + "CEP_CODE", # common compound form e.g. BRAZIL_CEP_CODE + "PLZ", # DE Postleitzahl + ], + "COUNTRY": ["LOCATION_COUNTRY", "COUNTRY_OR_REGION"], + }, + "BUILDING": [], + "FACILITY": [], + "GEO_COORDINATES": [ + "GPSCOORDINATES", + "GPS_COORDINATES", + "COORDINATE", + "LOCATION_COORDINATE", + "LATITUDE_LONGITUDE", + "NEARBYGPSCOORDINATE", + "GEOCOORD", + "LAT", + "LONG", + "LATITUDE", + "LONGITUDE", + ], + "LOCATION_OTHER": ["LOCATION-OTHER", "ORDINALDIRECTION"], + "GPE": ["GLOBAL_POLITICAL_ENTITY"], + "LOC": [], + "GEO": [], + }, + "ORGANIZATION": { + "COMPANY": [ + "COMPANYNAME", + "COMPANY_ID", + "COMPANY_NAME", + "CORPORATION", + "VENDOR", + "HOSPITAL", + ], + "GOVERNMENT_AGENCY": ["GOVERNMENT"], + "SCHOOL": ["SCHOOL_ID"], + "MEDICAL_FACILITY": [ + "ORGANIZATION_MEDICAL_FACILITY", + "HOSPITAL", + "HOSPITAL_NAME", + ], + "OTHER_ORG": [], + "ORG": [], + }, + "EMPLOYMENT": { + "JOB_TITLE": ["JOBTITLE", "OCCUPATION", "PROFESSION", "PROVIDER", "POSITION"], + "JOB_DEPARTMENT": ["JOBDEPARTMENT", "JOBAREA"], + "JOB_DESCRIPTOR": ["JOBDESCRIPTOR", "JOBTYPE"], + "EMPLOYEE_ID": ["EMPLOYEE"], + "CUSTOMER_ID": ["CUSTOMER", "UNIQUE", "UNIQUE_ID"], + "EMPLOYMENT_STATUS": [], + "LICENSE": [], + }, + "GOVERNMENT_ID": { + "SSN": [ + "SOCIALNUMBER", + "SOCIALNUM", + "SOCIAL_SECURITY", + "SOCIAL_SECURITY_NUMBER", + "US_SSN", + "UK_NINO", + # Common generic forms used as standalone labels + "SOCIAL_INSURANCE", + "NATIONAL_INSURANCE", + "INSURANCE_NUMBER", + ], + "PASSPORT": [ + "PSP", + "PASSPORT_NUMBER", + "PASSPORT_ID", + "US_PASSPORT", + "UK_PASSPORT", + ], + "DRIVER_LICENSE": [ + "DRIVERLICENSE", + "DRIVERLICENSENUM", + "DLN", + "DRIVERS_LICENSE", + "DRIVER_LICENSE_ID", + "US_DRIVER_LICENSE", + "IT_DRIVER_LICENSE", + "DRIVER", # generic suffix keyword (e.g. GERMANY_DRIVER_LICENSE) + ], + "TAX_ID": [ + "TAXNUM", + "US_ITIN", + "AU_TFN", + "IN_PAN", + "IN_GSTIN", + "ES_NIE", + "ES_NIF", + # Common country-specific tax codes used as standalone labels + "CPF", # BR Cadastro de Pessoas Físicas + "RFC", # MX Registro Federal de Contribuyentes + "RUT", # CL Rol Único Tributario + "NIT", # CO Número de Identificación Tributaria + ], + "NATIONAL_ID": [ + "IDCARD", + "IDCARDNUM", + "ID_NUM", + "IDNUM", + "IT_FISCAL_CODE", + "IT_IDENTITY_CARD", + "IN_AADHAAR", + "PL_PESEL", + "SG_NRIC_FIN", + "FI_PERSONAL_IDENTITY_CODE", + "KR_RRN", + "KR_FRN", + "KR_BRN", + "NG_NIN", + "TH_TNIN", + # Common country-specific ID codes used as standalone labels + "AADHAAR", # IN Aadhaar card + "DNI", # ES/AR/PE Documento Nacional de Identidad + "CITIZENSHIP_CARD", # CO Cédula de Ciudadanía + "CURP", # MX Clave Única de Registro de Población + "RG_NUMBER", # BR Registro Geral + "RUN", # CL Rol Único Nacional + "NATIONAL_IDENTIFICATION", # generic suffix (e.g. FRANCE_NATIONAL_IDENTIFICATION_NUMBER) + "RRN", # KR standalone Resident Registration Number + ], + "VOTER_ID": ["VOTER", "IN_VOTER", "UK_ELECTORAL_ROLL_NUMBER", "ELECTORAL"], + "IMMIGRATION_ID": ["IMMIGRATION"], + "PROFESSIONAL_LICENSE": [ + "LICENSE", + "CERTIFICATE_LICENSE_NUMBER", + "PROFESSIONAL_LICENSE_ID", + "IT_VAT_CODE", + ], + "BUSINESS_ID": [ + "BUSINESS", "SG_UEN", "AU_ABN", "AU_ACN", + "HANDELSREGISTER", # DE Handelsregisternummer + "REGISTRO_MERCANTIL", # ES/CO Registro Mercantil + "CNPJ", # BR Cadastro Nacional da Pessoa Jurídica + ], + "PUBLIC_TRANSPORT_CARD": [], + "ID": ["NUMERIC_PII", "CODE"], # CODE: generic coded identifier + "VIN": ["VIN_ID", "VEHICLE_IDENTIFICATION_NUMBER"], + "LICENSE_PLATE_NUMBER": [ + "LICENSE_PLATE_ID", + "VEHICLE_REGISTRATION_NUMBER", + "VRN", + "LICENSE_PLATE", + "KFZ", # DE Kraftfahrzeugkennzeichen + "KENNZEICHEN", # DE Kennzeichen + ], + }, + "FINANCIAL_PII": { + "FINANCIAL": { + "CREDIT_CARD": { + "CARD_NUMBER": [ + "CREDITCARD", + "CREDIT_CARD", + "CREDITCARDNUMBER", + "CREDIT_DEBIT_CARD", + "CRD", + ], + "CVV": ["CREDITCARDCVV", "CREDIT_CARD_SECURITY_CODE"], + "EXPIRATION": ["CREDIT_CARD_EXPIRATION"], + "CARD_ISSUER": ["CREDITCARDISSUER", "CARDISSUER"], + "MASKED_NUMBER": ["MASKEDNUMBER"], + }, + "BANK_ACCOUNT": { + "ACCOUNT_NUMBER": [ + "ACCOUNT", + "BANKACCOUNT", + "BANK_ACCOUNT", + "ACCOUNTNUMBER", + "ACCOUNTNUM", + "ACC", + "BBAN", + ], + "IBAN": ["IBAN_CODE"], + "SWIFT_BIC": ["BIC", "SWIFT_CODE"], + "ROUTING_NUMBER": ["BANK_ROUTING_NUMBER"], + }, + "CRYPTO_WALLET": { + "BITCOIN": ["BITCOINADDRESS", "CRYPTO"], + "ETHEREUM": ["ETHEREUMADDRESS"], + "LITECOIN": ["LITECOINADDRESS"], + }, + "FINANCIAL_AMOUNT": { + "CURRENCY": ["CURRENCYCODE", "CURRENCYNAME", "CURRENCYSYMBOL"], + "AMOUNT": ["MONEY"], + }, + "INSURANCE": { + "POLICY_NUMBER": ["INSURANCE_POLICY", "POLICY_ID"], + "CLAIM_NUMBER": ["CLAIM_ID", "INSURANCE_CLAIM"], + "POLICY_HOLDER": [], + }, + }, + }, + "DEVICE_IDENTIFIER": { + "DEVICE_ID": ["DEVICE", "DEVICE_IDENTIFIER"], + "SERIAL_NUMBER": [], + "IMEI": ["PHONEIMEI"], + "IMSI": ["SUBSCRIBER_IDENTITY", "MOBILE_SUBSCRIBER_ID"], + "ICCID": ["SIM_CARD_NUMBER", "SIM_ID"], + "MAC_ADDRESS": ["MACADDRESS", "MAC"], + "ADVERTISING_ID": ["ADVERTISING"], + "USER_AGENT": ["USERAGENT"], + "FILE_PATH": ["FILENAME"], # file/path reference that may contain PII + }, + "BIOMETRIC": { + "FINGERPRINT": [ + "FINGERPRINT_ID", + "FINGERPRINT_DATA", + "FINGERPRINT_TEMPLATE", + ], + "FACE": [ + "FACE_ID", + "FACE_RECOGNITION", + "FACIAL_SCAN", + "FACE_TEMPLATE", + "BIOID", + "BIOMETRIC_IDENTIFIER", + "PHOTO", + "FACIAL_IMAGE", + "FACE_IMAGE", + ], + "IRIS": ["IRIS_SCAN", "IRIS_TEMPLATE"], + "RETINA": ["RETINA_SCAN"], + "VOICE_PRINT": ["VOICE_RECOGNITION", "VOICE_TEMPLATE", "VOICEPRINT"], + "DNA": ["DNA_SEQUENCE", "GENETIC_DATA"], + "PALM_PRINT": ["PALM_TEMPLATE", "PALM_VEIN"], + }, + "NETWORK_IDENTIFIER": { + "IP_ADDRESS": ["IPADDRESS", "IP", "IPV4", "IPV6"], + "URL": [], + "DOMAIN": [], + "HTTP_COOKIE": ["COOKIE_ID", "HTTP_COOKIE_ID"], + "CONNECTION_STRING": [], + }, + "AUTHENTICATION": { + "PASSWORD": ["PASS", "PWD"], + "PIN": [], + "API_KEY": [], + "PRIVATE_KEY": [], + "TOKEN": ["SECURITYTOKEN"], + }, + "PHI": { + "PATIENT_ID": [ + "PATIENT", + "MRN", + "MEDICALRECORD", + "MEDICAL_RECORD_NUMBER", + "MEDICAL_RECORD", + ], + "HEALTH_INSURANCE_ID": [ + "HEALTH_INSURANCE", + "HEALTH_PLAN", + "HEALTHPLAN", + "HEALTHCARE_NUMBER", + "HEALTH_PLAN_BENEFICIARY_NUMBER", + "UK_NHS", + "AU_MEDICARE", + "KVNR", # DE Krankenversicherungsnummer + "KRANKENVERSICHERUNG", # DE full word form + ], + "MEDICAL_LICENSE": ["MEDICAL_LICENSE_ID", "US_NPI", "US_MBI"], + "HEALTH_CONDITION": [ + "CONDITION", + "MEDICAL_DISEASE_DISORDER", + "MEDICAL_BIOLOGICAL_ATTRIBUTE", + "MEDICAL_BIOLOGICAL_STRUCTURE", + ], + "MEDICATION": ["DRUG", "DOSE", "MEDICAL_MEDICATION"], + "PROCEDURE": [ + "MEDICAL_PROCESS", + "MEDICAL_THERAPEUTIC_PROCEDURE", + "MEDICAL_CLINICAL_EVENT", + ], + "INJURY": [], + "BLOOD_TYPE": [], + "FAMILY_HISTORY": ["MEDICAL_FAMILY_HISTORY", "MEDICAL_HISTORY"], + "STATISTICS": [], + "MRN": ["MEDICAL_RECORD_NUMBER"], + "PLAN": ["HEALTHCARE_PLAN"], + "PROTECTED_HEALTH_INFORMATION": [], + # Clinical research / trials + "STUDY_PARTICIPANT_ID": ["SUBJECT_ID", "TRIAL_PARTICIPANT_ID", "PARTICIPANT_ID"], + "PROTOCOL_ID": ["IRB_NUMBER", "PROTOCOL_NUMBER", "STUDY_ID"], + "COHORT_ID": ["COHORT", "ARM_ID"], + }, + "VEHICLE_PII": { + "LICENSE_PLATE": [ + "VRM", + "VEHICLE_IDENTIFIER", + "VEHICLE_REGISTRATION", + "VRN", + ], + "VIN": ["VEHICLEVIN", "VEHICLE_IDENTIFICATION_NUMBER"], + "VEHICLE_ID": ["VEHICLEVRM"], + "CAR_TYPE": ["VEHICLE_TYPE", "VEHICLE_MAKE", "MAKE_MODEL"], + }, + "LEGAL_PII": { + "CASE_NUMBER": [], + "COURT_RECORD": [], + "ARREST_RECORD": [], + "INMATE_ID": ["INMATE"], + "MISCELLANEOUS": ["MISC"], # catch-all for unclassified NER labels + }, + "TRAVEL_PII": { + "PNR": ["PASSENGER_NAME_RECORD", "PNR_NUMBER"], + "ETIX": ["ELECTRONIC_TICKET", "ETICKET"], + "WTN": ["WTN_NUMBER", "WORLD_TRACER_NUMBER"], + }, + "EDUCATION": { + "STUDENT_ID": ["STUDENT_NUMBER", "LEARNER_ID", "ENROLLMENT_NUMBER", "STUDENT"], + "ACADEMIC_RECORD": ["TRANSCRIPT", "GRADE_REPORT", "GPA"], + "EDUCATION_LEVEL": [], + "INSTITUTION_ID": ["SCHOOL_CODE", "UNIVERSITY_ID"], + "PARENT_GUARDIAN_ID": [], + "PARENT": [], + "TEACHER_ID": ["TEACHER_NUMBER", "FACULTY_ID", "TEACHER", "FACULTY"], + + }, + "DATE_TIME": { + "DATE": [], + "TIME": [], + "EPOCH": [], + "BIRTH_DATE": ["DATEOFBIRTH", "DATE_OF_BIRTH", "DOB", "BIRTHDAY", "BOD"], + "DEATH_DATE": [], + "DATE_INTERVAL": [], + "DURATION": [], + "EVENT": [], + "DATES": [], + }, + } +} +``` From e1b588414cb173e74ca88ecb5a3c542f813f9ed4 Mon Sep 17 00:00:00 2001 From: Omri Mendels Date: Sat, 21 Mar 2026 17:08:08 +0200 Subject: [PATCH 07/22] Update entity mapping details in ADR-002 Clarified the hierarchical entity mapping dictionary with specific examples for 2nd and 3rd levels. --- docs/adr/ADR-002-entity-mapping.md | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/docs/adr/ADR-002-entity-mapping.md b/docs/adr/ADR-002-entity-mapping.md index 6d201633..550f1271 100644 --- a/docs/adr/ADR-002-entity-mapping.md +++ b/docs/adr/ADR-002-entity-mapping.md @@ -165,9 +165,11 @@ and allows the same model output to be evaluated under different mapping configu ## Proposed hierarchical entity mapping dictionary All entities are mapped to the 3rd level (canonoical) -2nd level: PERSON, DEMOGRAPHIC, CONTACT, LOCATION, ORGANIZATION, EMPLOYMENT, GOVERNMENT_ID, FINANCIAL_PII, DEVICE_IDENTIFIER, BIOMETRIC, NETWORK_IDENTIFIER, AUTHENTICATION, PHI, VEHICLE_PII, LEGAL_PII, TRAVEL_PII, EDUCATION, DATE_TIME -3rd level: NAME, ..., TITLE, USERNAME, ..., AGE, GENDER, ..., ADDRESS, ..., COMPANY, -Although the user can choose a different level of granularity (coarser or broader +- The 2nd level: `PERSON, DEMOGRAPHIC, CONTACT, LOCATION, ORGANIZATION, EMPLOYMENT, GOVERNMENT_ID, FINANCIAL_PII, DEVICE_IDENTIFIER, BIOMETRIC, NETWORK_IDENTIFIER, AUTHENTICATION, PHI, VEHICLE_PII, LEGAL_PII, TRAVEL_PII, EDUCATION, DATE_TIME`. +- The 3rd level: `NAME, ..., TITLE, USERNAME, ..., AGE, GENDER, ..., ADDRESS, ..., COMPANY, SSN, PASSPORT, TAX_ID, NATIONAL_ID, FINANCIAL, DEVICE_ID,...` + + +The user can choose a different level of granularity (coarser or broader), add a new mapping or change mappings. ```py HIERARCHY: dict = { From ec19dc7e0353e87a3b8d9093752ff989b269ef9a Mon Sep 17 00:00:00 2001 From: Omri Mendels Date: Sat, 21 Mar 2026 17:11:21 +0200 Subject: [PATCH 08/22] Refactor CanonicalMapper usage examples Removed interactive resolution and clarified mapping retrieval process. --- docs/adr/ADR-002-entity-mapping.md | 19 ++++++------------- 1 file changed, 6 insertions(+), 13 deletions(-) diff --git a/docs/adr/ADR-002-entity-mapping.md b/docs/adr/ADR-002-entity-mapping.md index 550f1271..28bc7205 100644 --- a/docs/adr/ADR-002-entity-mapping.md +++ b/docs/adr/ADR-002-entity-mapping.md @@ -64,14 +64,9 @@ CanonicalMapper(labels) → auto-resolve pass (EXACT → COUNTRY → COUNTRY_FALLBACK → FUZZY) → inspect .pending # labels that need attention → .map({...}) # programmatic assignment - → .resolve_interactively() # guided terminal/notebook prompts → .get_mapping() # returns dict[str, str | None] ``` -When all labels auto-resolve, `.get_mapping()` can be called immediately — no additional steps -are needed. When pending labels remain, `.get_mapping()` raises `IncompleteMapping` until every -label is accounted for. - ### Typical usage ```python @@ -82,16 +77,14 @@ label is accounted for. # .get_mapping(). # Everything auto-resolves — mapping dict returned immediately -mapping = CanonicalMapper.from_dataset(samples) +dataset_mapping = CanonicalMapper.from_dataset(samples) +model_mapping = CanonicalMapper.from_model(huggingface_model) #or PresidioAnalyzer -# Some labels are pending — resolve interactively, then retrieve -mapper = CanonicalMapper(["PERSON", "EMAIL_ADDRESS", "MY_CUSTOM_LABEL"]) -mapper.resolve_interactively() -mapping = mapper.get_mapping() +# If some labels are not mapped — update manually +mapper = CanonicalMapper.update({"GGE":"ORG", "CustID":"CLIENT_ID", "MY_CUSTOM_LABEL":None}) -# Programmatic (batch) assignment — no prompts -mapper.map({"MY_CUSTOM_LABEL": "PERSON", "INTERNAL_ID": None}) -mapping = mapper.get_mapping() +# Assuming all entities are now mapped: +mapping: Dict[str, str] = mapper.get_mapping() ``` ### Logging From 87a9d8e14aef5a2d19869ee3051ff068c0768e2d Mon Sep 17 00:00:00 2001 From: Omri Mendels Date: Sun, 22 Mar 2026 08:06:20 +0200 Subject: [PATCH 09/22] Update docs/adr/ADR-001-simplified-evaluation-pipeline.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --- docs/adr/ADR-001-simplified-evaluation-pipeline.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/adr/ADR-001-simplified-evaluation-pipeline.md b/docs/adr/ADR-001-simplified-evaluation-pipeline.md index 6ab8fc8d..47914f04 100644 --- a/docs/adr/ADR-001-simplified-evaluation-pipeline.md +++ b/docs/adr/ADR-001-simplified-evaluation-pipeline.md @@ -27,7 +27,7 @@ This design has four concrete pain points: 2. **`evaluate_all()` does two things** — it runs model inference AND builds per-sample `EvaluationResult` objects. These objects are simple data carriers holding `(tokens, actual_tags, predicted_tags, start_indices)`, yet they require callers to go through the evaluator just to get predictions into a usable shape. -3. **The real interface already exists but is buried** — `SpanEvaluator.calculate_score()` and `TokenEvaluator.calculate_score()` internally call `get_results_dataframe()` to convert `List[EvaluationResult]` to a DataFrame, and then call `calculate_score_on_df()`. The `EvaluationResult` list is a wasteful intermediate step; the DataFrame is the actual computational surface. +3. **The real interface already exists but is buried** — for span-level evaluation, `SpanEvaluator.calculate_score()` internally calls `get_results_dataframe()` to convert `List[EvaluationResult]` to a DataFrame, and then calls `SpanEvaluator.calculate_score_on_df()`. The `EvaluationResult` list is a wasteful intermediate step; the DataFrame is the actual computational surface. 4. **Entity mapping logic is scattered** — entity type remapping lives in `BaseModel.align_entity_types`, `BaseEvaluator.align_entity_types` (static method), and is applied independently in both models and evaluators. There is no single, composable entry point for this transformation. From cddbdda7cd592122cc4d960abe957ed1d74f09fb Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Sun, 22 Mar 2026 06:38:24 +0000 Subject: [PATCH 10/22] docs: move DOMAIN, WEB, URL, WEBSITE to NETWORK_IDENTIFIER Co-authored-by: omri374 <3776619+omri374@users.noreply.github.com> Agent-Logs-Url: https://github.com/microsoft/presidio-research/sessions/0cb7a127-de0a-4167-a2cb-a4b98a8f4028 --- docs/adr/ADR-002-entity-mapping.md | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/docs/adr/ADR-002-entity-mapping.md b/docs/adr/ADR-002-entity-mapping.md index 28bc7205..a157400f 100644 --- a/docs/adr/ADR-002-entity-mapping.md +++ b/docs/adr/ADR-002-entity-mapping.md @@ -229,8 +229,6 @@ HIERARCHY: dict = { "PHONE_NUMBER": ["PHONE", "TEL", "TELEPHONENUM", "PHONENUMBER", "PHN", "MOBILE"], "FAX": ["FAX_NUMBER"], "SOCIAL_HANDLE": ["QQ"], # QQ: Chinese messaging platform ID - # URL lives only under NETWORK_IDENTIFIER to avoid ambiguity - "WEBSITE": ["DOMAIN", "WEB"], }, "LOCATION": { "ADDRESS": { @@ -483,8 +481,9 @@ HIERARCHY: dict = { }, "NETWORK_IDENTIFIER": { "IP_ADDRESS": ["IPADDRESS", "IP", "IPV4", "IPV6"], - "URL": [], - "DOMAIN": [], + "URL": ["URI", "HYPERLINK"], + "DOMAIN": ["DOMAINNAME", "DOMAIN_NAME"], + "WEBSITE": ["WEB", "WEBPAGE", "WEBADDRESS"], "HTTP_COOKIE": ["COOKIE_ID", "HTTP_COOKIE_ID"], "CONNECTION_STRING": [], }, From e0daa3cccd261b37e4a5fce6fe8625c2033e4e4c Mon Sep 17 00:00:00 2001 From: Omri Mendels Date: Sun, 22 Mar 2026 09:25:00 +0200 Subject: [PATCH 11/22] Update docs/adr/ADR-001-simplified-evaluation-pipeline.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --- docs/adr/ADR-001-simplified-evaluation-pipeline.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/adr/ADR-001-simplified-evaluation-pipeline.md b/docs/adr/ADR-001-simplified-evaluation-pipeline.md index 47914f04..1fc512f4 100644 --- a/docs/adr/ADR-001-simplified-evaluation-pipeline.md +++ b/docs/adr/ADR-001-simplified-evaluation-pipeline.md @@ -88,7 +88,7 @@ plotter.plot_scores() - **Decoupled model and evaluator** — models and evaluators can be developed, tested, and swapped independently. `SpanEvaluator` and `TokenEvaluator` no longer require a `model` instance. - **Composable entity mapping** — `map_entities` is a pure function on DataFrames; it can be chained, applied selectively, or omitted without touching model or evaluator configuration. - **Easier offline/batch evaluation** — a results DataFrame can be saved to disk and re-evaluated later without re-running inference. This is currently impossible without serializing `List[EvaluationResult]` objects. -- **`calculate_score_on_df` already exists and is tested** — `SpanEvaluator.calculate_score_on_df` and `TokenEvaluator.calculate_score_on_df` are already implemented and covered by tests in `test_span_evaluator.py`. The new pipeline plugs directly into existing code. +- **`calculate_score_on_df` already exists and is tested for spans** — `SpanEvaluator.calculate_score_on_df` is already implemented and covered by tests in `test_span_evaluator.py`. A similar API can be added for `TokenEvaluator` as part of this ADR. - **Eliminates the `EvaluationResult` per-sample carrier** — removes a class whose only purpose is to ferry data between `evaluate_all()` and `get_results_dataframe()`. ### Negative / Trade-offs From d7d95e9cfb205099f3c79ddafc5ad5f0675903e4 Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Sun, 22 Mar 2026 07:26:15 +0000 Subject: [PATCH 12/22] docs: update CanonicalMapper API and map_entities interface per review feedback Co-authored-by: omri374 <3776619+omri374@users.noreply.github.com> Agent-Logs-Url: https://github.com/microsoft/presidio-research/sessions/8617ff64-a48a-4bc1-96cb-e83ac8d52192 --- .../ADR-001-simplified-evaluation-pipeline.md | 3 +- docs/adr/ADR-002-entity-mapping.md | 30 +++++++++---------- 2 files changed, 17 insertions(+), 16 deletions(-) diff --git a/docs/adr/ADR-001-simplified-evaluation-pipeline.md b/docs/adr/ADR-001-simplified-evaluation-pipeline.md index 1fc512f4..ca786210 100644 --- a/docs/adr/ADR-001-simplified-evaluation-pipeline.md +++ b/docs/adr/ADR-001-simplified-evaluation-pipeline.md @@ -58,7 +58,8 @@ model = PresidioAnalyzerWrapper(model=AnalyzerEngine()) results_df = model.predict_dataset(dataset) # NEW: returns the DataFrame directly # 3. Map entities (transforms both predictions and annotations into canonical entities) -results_df = map_entities(results_df) +# ENTITY_MAPPING can be defined in config or as a shared constant. +results_df = map_entities(results_df, mapping=ENTITY_MAPPING) # 4. Score evaluator = SpanEvaluator() # no model argument needed! diff --git a/docs/adr/ADR-002-entity-mapping.md b/docs/adr/ADR-002-entity-mapping.md index a157400f..6c521f87 100644 --- a/docs/adr/ADR-002-entity-mapping.md +++ b/docs/adr/ADR-002-entity-mapping.md @@ -60,30 +60,30 @@ all outputs. ### Workflow ``` -CanonicalMapper(labels) +CanonicalMapper.from_dataset(dataset) / CanonicalMapper.from_model(label_extractor) → auto-resolve pass (EXACT → COUNTRY → COUNTRY_FALLBACK → FUZZY) - → inspect .pending # labels that need attention - → .map({...}) # programmatic assignment - → .get_mapping() # returns dict[str, str | None] + → mapper.get_mapping(mode='html') # inspect gaps as an HTML table + → mapper.map({...}) # programmatic assignment for pending labels + → mapper.get_mapping() # returns dict[str, str | None] ``` ### Typical usage ```python -# `from_dataset` is a convenience constructor: it extracts entity labels from the -# dataset's spans, runs the auto-resolve pass, and returns the mapping dict directly -# if every label resolves automatically. When labels are still pending it returns -# the CanonicalMapper instance so the caller can resolve them before calling -# .get_mapping(). +# Construct the mapper from a dataset or a model. +# Both constructors extract the label vocabulary, run the auto-resolve pass, +# and return a CanonicalMapper instance. +mapper = CanonicalMapper.from_dataset(dataset) # dataset: List[InputSample] | pd.DataFrame +# or +mapper = CanonicalMapper.from_model(label_extractor) # label_extractor: LabelExtractor -# Everything auto-resolves — mapping dict returned immediately -dataset_mapping = CanonicalMapper.from_dataset(samples) -model_mapping = CanonicalMapper.from_model(huggingface_model) #or PresidioAnalyzer +# Inspect unmapped labels — renders an HTML table highlighting gaps +mapper.get_mapping(mode='html') -# If some labels are not mapped — update manually -mapper = CanonicalMapper.update({"GGE":"ORG", "CustID":"CLIENT_ID", "MY_CUSTOM_LABEL":None}) +# Resolve pending labels manually +mapper.map({"GGE": "ORG", "CustID": "CLIENT_ID", "MY_CUSTOM_LABEL": None}) -# Assuming all entities are now mapped: +# Retrieve the final mapping dict once all labels are resolved mapping: Dict[str, str] = mapper.get_mapping() ``` From 6d6866f03ff45b59ee9dc1be95fb1a8ddf3d67ee Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Sun, 22 Mar 2026 07:42:06 +0000 Subject: [PATCH 13/22] docs: remove HOSPITAL alias from ORGANIZATION.COMPANY Co-authored-by: omri374 <3776619+omri374@users.noreply.github.com> Agent-Logs-Url: https://github.com/microsoft/presidio-research/sessions/753f1074-cc0d-445e-b838-50b11eb55356 --- docs/adr/ADR-002-entity-mapping.md | 1 - 1 file changed, 1 deletion(-) diff --git a/docs/adr/ADR-002-entity-mapping.md b/docs/adr/ADR-002-entity-mapping.md index 6c521f87..2d7124fa 100644 --- a/docs/adr/ADR-002-entity-mapping.md +++ b/docs/adr/ADR-002-entity-mapping.md @@ -283,7 +283,6 @@ HIERARCHY: dict = { "COMPANY_NAME", "CORPORATION", "VENDOR", - "HOSPITAL", ], "GOVERNMENT_AGENCY": ["GOVERNMENT"], "SCHOOL": ["SCHOOL_ID"], From 151acad610b31da0ee43d55ddb99b3b6c160d8d6 Mon Sep 17 00:00:00 2001 From: Omri Mendels Date: Sun, 22 Mar 2026 09:42:13 +0200 Subject: [PATCH 14/22] Apply suggestions from code review Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --- .../ADR-001-simplified-evaluation-pipeline.md | 8 ++++---- docs/adr/ADR-002-entity-mapping.md | 17 +++++++++-------- 2 files changed, 13 insertions(+), 12 deletions(-) diff --git a/docs/adr/ADR-001-simplified-evaluation-pipeline.md b/docs/adr/ADR-001-simplified-evaluation-pipeline.md index ca786210..706e341e 100644 --- a/docs/adr/ADR-001-simplified-evaluation-pipeline.md +++ b/docs/adr/ADR-001-simplified-evaluation-pipeline.md @@ -23,7 +23,7 @@ Dataset (List[InputSample]) This design has four concrete pain points: -1. **Model is coupled to the Evaluator** — `BaseEvaluator.__init__` takes a `model` argument, and `evaluate_all()` calls `model.batch_predict`, `model.filter_tags_in_supported_entities`, and `model.to_scheme` internally. This makes it impossible to evaluate a pre-computed result set without also instantiating a model. +1. **Model is coupled to the Evaluator** — `BaseEvaluator.__init__` takes a `model` argument, and `evaluate_all()` calls `model.batch_predict`, `model.filter_tags_in_supported_entities`, and `model.to_scheme` internally. While it is technically possible to evaluate a pre-computed result set via `SpanEvaluator(model=None)` and `calculate_score_on_df()` on a results DataFrame, this coupling makes that path non-obvious and discourages treating the DataFrame-based interface as a first-class entry point. 2. **`evaluate_all()` does two things** — it runs model inference AND builds per-sample `EvaluationResult` objects. These objects are simple data carriers holding `(tokens, actual_tags, predicted_tags, start_indices)`, yet they require callers to go through the evaluator just to get predictions into a usable shape. @@ -54,7 +54,7 @@ This schema is already produced by `get_results_dataframe()` and consumed by `ca dataset = InputSample.read_dataset_json("data/dataset.json") # 2. Choose model and run predictions → get DataFrame directly -model = PresidioAnalyzerWrapper(model=AnalyzerEngine()) +model = PresidioAnalyzerWrapper(analyzer_engine=AnalyzerEngine()) results_df = model.predict_dataset(dataset) # NEW: returns the DataFrame directly # 3. Map entities (transforms both predictions and annotations into canonical entities) @@ -105,11 +105,11 @@ plotter.plot_scores() 2. **Add `map_entities()` utility** — add the function (and `Dict` import) to `presidio_evaluator/evaluation/` (e.g., in a new `utils.py` or alongside `get_results_dataframe`). Add a unit test verifying that both `annotation` and `prediction` columns are remapped. -3. **Make `model` optional in `BaseEvaluator`** — change `BaseEvaluator.__init__(self, model, ...)` so that `model` defaults to `None`. Add a runtime check that raises a clear error if `evaluate_all()` is called when `model is None`. +3. **Make `model` optional in `BaseEvaluator`** — change `BaseEvaluator.__init__(self, model=None, ...)` so that `model` defaults to `None`, relying on the existing runtime check in `evaluate_all()` that raises a clear error when `model is None`. 4. **Update `evaluate_all()` to delegate to `predict_dataset` + `calculate_score_on_df`** — refactor `SpanEvaluator.evaluate_all()` and `TokenEvaluator.evaluate_all()` to call `self.model.predict_dataset(dataset)` and then pass the result to `calculate_score_on_df()`. This ensures a single code path for both old and new usage. -5. **Deprecate per-sample `EvaluationResult`** — add a `DeprecationWarning` to the `EvaluationResult` class (the per-sample variant) and update `get_results_dataframe()` to note it will be removed in a future release. +5. **Deprecate the per-sample *usage* of `EvaluationResult`** — keep `EvaluationResult` as the aggregated-metrics return type, but stop using it as an intermediate per-sample carrier. Mark the per-sample–only fields/usages (e.g., the path that produces `List[EvaluationResult]` for individual samples) with a `DeprecationWarning`, and update `get_results_dataframe()` docs to note that this per-sample usage will be removed in a future release and replaced by the DataFrame-based pipeline (or, if needed, a dedicated `SampleEvaluationResult` type introduced later). 6. **Update documentation and notebooks** — revise `docs/evaluation.md` and any Jupyter notebooks under `notebooks/` to demonstrate the new 5-step pipeline. Keep the old `evaluate_all()` example with a note that it is the backward-compatible convenience path. diff --git a/docs/adr/ADR-002-entity-mapping.md b/docs/adr/ADR-002-entity-mapping.md index 2d7124fa..ebe86780 100644 --- a/docs/adr/ADR-002-entity-mapping.md +++ b/docs/adr/ADR-002-entity-mapping.md @@ -26,13 +26,14 @@ The current approach has two pain points: ## Decision Introduce a single, stateful class — `CanonicalMapper` — as the sole entry point for resolving -user-defined labels to **canonical entities**. A canonical entity is a normalised, taxonomy-defined -label drawn from the `EntityHierarchy` vocabulary (e.g. `PERSON`, `EMAIL_ADDRESS`, -`PASSPORT`). Mapping a raw label to a canonical entity means finding the single taxonomy entry that -best represents the concept the raw label describes. Once every label on both sides of an evaluation -— dataset annotations and model predictions — has been mapped to a canonical entity, scores can be -computed fairly: identical canonical entities count as matches regardless of how the two sides -originally spelled or tagged them. +user-defined labels to **canonical entities**. In this ADR, a canonical entity is a normalised, +taxonomy-defined label corresponding to a 3rd-level (leaf) node in the `EntityHierarchy` +vocabulary (e.g. `NAME`, `AGE`, `ADDRESS`). Mapping a raw label to a canonical entity means +finding the single 3rd-level taxonomy entry that best represents the concept the raw label +describes. Once every label on both sides of an evaluation — dataset annotations and model +predictions — has been mapped to a canonical entity, scores can be computed fairly: identical +canonical entities count as matches regardless of how the two sides originally spelled or tagged +them. The same class is used for both sides of an evaluation: the dataset's label vocabulary and the model's output label vocabulary. Both are resolved independently; evaluation then compares @@ -157,7 +158,7 @@ and allows the same model output to be evaluated under different mapping configu ## Proposed hierarchical entity mapping dictionary -All entities are mapped to the 3rd level (canonoical) +All entities are mapped to the 3rd level (canonical) - The 2nd level: `PERSON, DEMOGRAPHIC, CONTACT, LOCATION, ORGANIZATION, EMPLOYMENT, GOVERNMENT_ID, FINANCIAL_PII, DEVICE_IDENTIFIER, BIOMETRIC, NETWORK_IDENTIFIER, AUTHENTICATION, PHI, VEHICLE_PII, LEGAL_PII, TRAVEL_PII, EDUCATION, DATE_TIME`. - The 3rd level: `NAME, ..., TITLE, USERNAME, ..., AGE, GENDER, ..., ADDRESS, ..., COMPANY, SSN, PASSPORT, TAX_ID, NATIONAL_ID, FINANCIAL, DEVICE_ID,...` From 44adbc0e0cd5892133ddc84d26b3f463f4fb32bf Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Sun, 22 Mar 2026 07:43:41 +0000 Subject: [PATCH 15/22] docs: remove MRN alias from PHI.PATIENT_ID Co-authored-by: omri374 <3776619+omri374@users.noreply.github.com> Agent-Logs-Url: https://github.com/microsoft/presidio-research/sessions/6abd899f-aafd-4d21-bbde-c535cfc20d6e --- docs/adr/ADR-002-entity-mapping.md | 1 - 1 file changed, 1 deletion(-) diff --git a/docs/adr/ADR-002-entity-mapping.md b/docs/adr/ADR-002-entity-mapping.md index ebe86780..ad9c4542 100644 --- a/docs/adr/ADR-002-entity-mapping.md +++ b/docs/adr/ADR-002-entity-mapping.md @@ -497,7 +497,6 @@ HIERARCHY: dict = { "PHI": { "PATIENT_ID": [ "PATIENT", - "MRN", "MEDICALRECORD", "MEDICAL_RECORD_NUMBER", "MEDICAL_RECORD", From a56c6ac5d5327d8fffce444c8dff44476839273a Mon Sep 17 00:00:00 2001 From: Omri Mendels Date: Sun, 22 Mar 2026 22:27:48 +0200 Subject: [PATCH 16/22] Add example mappings for entity labels Added example mappings of raw entity labels to canonical entities in the EntityHierarchy, enhancing clarity on how different labels correspond to standardized entities. --- docs/adr/ADR-002-entity-mapping.md | 249 ++++++++++++++++++++++++++++- 1 file changed, 248 insertions(+), 1 deletion(-) diff --git a/docs/adr/ADR-002-entity-mapping.md b/docs/adr/ADR-002-entity-mapping.md index ad9c4542..d0f53200 100644 --- a/docs/adr/ADR-002-entity-mapping.md +++ b/docs/adr/ADR-002-entity-mapping.md @@ -155,7 +155,254 @@ rejected because mapping is a property of the *evaluation goal* (which canonical score), not of the model. Keeping mapping separate makes both components independently testable and allows the same model output to be evaluated under different mapping configurations. - +## Example mappings + +Mapping of raw entity labels (as found in HuggingFace NER models and datasets) to canonical entities in the `EntityHierarchy`. + +| Raw Label | Canonical Entity | +|-----------|-----------------| +| `ACCOUNT` | `FINANCIAL` | +| `ACCOUNTNAME` | `USERNAME` | +| `ACCOUNTNUM` | `FINANCIAL` | +| `ACCOUNTNUMBER` | `FINANCIAL` | +| `ACCOUNT_NUMBER` | `FINANCIAL` | +| `ADDRESS` | `ADDRESS` | +| `AGE` | `AGE` | +| `AMOUNT` | `FINANCIAL` | +| `BANKACCOUNT` | `FINANCIAL` | +| `BIC` | `FINANCIAL` | +| `BIOID` | `FACE` | +| `BIOMETRIC` | `BIOMETRIC` | +| `BIRTHDAY` | `BIRTH_DATE` | +| `BITCOINADDRESS` | `FINANCIAL` | +| `BOD` | `BIRTH_DATE` | +| `BUILDING` | `BUILDING` | +| `BUILDINGNUM` | `ADDRESS` | +| `BUILDINGNUMBER` | `ADDRESS` | +| `BUILDING_NUMBER` | `ADDRESS` | +| `CARDISSUER` | `FINANCIAL` | +| `CITY` | `ADDRESS` | +| `CNPJ` | `BUSINESS_ID` | +| `COMPANYNAME` | `COMPANY` | +| `COMPANY_NAME` | `COMPANY` | +| `CONDITION` | `HEALTH_CONDITION` | +| `CONT` | `ADDRESS` | +| `COORDINATE` | `GEO_COORDINATES` | +| `COUNTRY` | `ADDRESS` | +| `COUNTY` | `ADDRESS` | +| `CRD` | `FINANCIAL` | +| `CREDITCARD` | `FINANCIAL` | +| `CREDITCARDCVV` | `FINANCIAL` | +| `CREDITCARDISSUER` | `FINANCIAL` | +| `CREDITCARDNUMBER` | `FINANCIAL` | +| `CREDIT_CARD` | `FINANCIAL` | +| `CREDIT_CARD_NUMBER` | `FINANCIAL` | +| `CURRENCY` | `FINANCIAL` | +| `CURRENCYCODE` | `FINANCIAL` | +| `CURRENCYNAME` | `FINANCIAL` | +| `CURRENCYSYMBOL` | `FINANCIAL` | +| `CVV` | `FINANCIAL` | +| `DATE` | `DATE` | +| `DATEOFBIRTH` | `BIRTH_DATE` | +| `DATE_OF_BIRTH` | `BIRTH_DATE` | +| `DATE_TIME` | `DATE_TIME` | +| `DEVICE` | `DEVICE_ID` | +| `DLN` | `DRIVER_LICENSE` | +| `DOB` | `BIRTH_DATE` | +| `DOCTOR` | `NAME` | +| `DOCTOR_NAME` | `NAME` | +| `DRIVERLICENSE` | `DRIVER_LICENSE` | +| `DRIVERLICENSENUM` | `DRIVER_LICENSE` | +| `DRIVER_LICENSE_NUMBER` | `DRIVER_LICENSE` | +| `EMA` | `EMAIL_ADDRESS` | +| `EMAIL` | `EMAIL_ADDRESS` | +| `EMAIL_ADDRESS` | `EMAIL_ADDRESS` | +| `ETHEREUMADDRESS` | `FINANCIAL` | +| `EYECOLOR` | `PHYSICAL_DESCRIPTOR` | +| `FACILITY` | `FACILITY` | +| `FAX` | `FAX` | +| `FINANCIAL` | `FINANCIAL` | +| `FIRSTNAME` | `NAME` | +| `FIRST_NAME` | `NAME` | +| `FULLNAME` | `NAME` | +| `GENDER` | `GENDER` | +| `GEOCOORD` | `GEO_COORDINATES` | +| `GIVENNAME` | `NAME` | +| `GIVENNAME1` | `NAME` | +| `GIVENNAME2` | `NAME` | +| `GPSCOORDINATES` | `GEO_COORDINATES` | +| `HEALTHPLAN` | `HEALTH_INSURANCE_ID` | +| `HEALTH_PLAN` | `HEALTH_INSURANCE_ID` | +| `HEIGHT` | `PHYSICAL_DESCRIPTOR` | +| `HOSPITAL` | `MEDICAL_FACILITY` | +| `HOSPITAL_NAME` | `MEDICAL_FACILITY` | +| `IBAN` | `FINANCIAL` | +| `IBAN_CODE` | `FINANCIAL` | +| `ID` | `ID` | +| `IDCARD` | `NATIONAL_ID` | +| `IDCARDNUM` | `NATIONAL_ID` | +| `IDNUM` | `NATIONAL_ID` | +| `ID_CARD_NUMBER` | `FINANCIAL` | +| `IMEI` | `IMEI` | +| `IPADDRESS` | `IP_ADDRESS` | +| `IPV4` | `IP_ADDRESS` | +| `IPV6` | `IP_ADDRESS` | +| `IP_ADDRESS` | `IP_ADDRESS` | +| `JOBAREA` | `JOB_DEPARTMENT` | +| `JOBDEPARTMENT` | `JOB_DEPARTMENT` | +| `JOBDESCRIPTOR` | `JOB_DESCRIPTOR` | +| `JOBTITLE` | `JOB_TITLE` | +| `JOBTYPE` | `JOB_DESCRIPTOR` | +| `LASTNAME` | `NAME` | +| `LASTNAME1` | `NAME` | +| `LASTNAME2` | `NAME` | +| `LASTNAME3` | `NAME` | +| `LAST_NAME` | `NAME` | +| `LICENSE` | `PROFESSIONAL_LICENSE` | +| `LICENSEPLATENUM` | `LICENSE_PLATE_NUMBER` | +| `LICENSE_PLATE` | `LICENSE_PLATE` | +| `LITECOINADDRESS` | `FINANCIAL` | +| `LOC` | `LOC` | +| `LOCATION` | `LOCATION` | +| `LOCATION-OTHER` | `LOCATION_OTHER` | +| `MAC` | `MAC_ADDRESS` | +| `MACADDRESS` | `MAC_ADDRESS` | +| `MAC_ADDRESS` | `MAC_ADDRESS` | +| `MASKEDNUMBER` | `FINANCIAL` | +| `MEDICALRECORD` | `PATIENT_ID` | +| `MIDDLENAME` | `NAME` | +| `MISC` | `MISCELLANEOUS` | +| `MRN` | `MRN` | +| `NAME` | `NAME` | +| `NATIONALID` | `NATIONAL_ID` | +| `NEARBYGPSCOORDINATE` | `GEO_COORDINATES` | +| `NO_RESPONSE` | `NATIONAL_ID` | +| `NRP` | `NATIONALITY` | +| `NUMBER` | `NATIONAL_ID` | +| `OCCUPATION` | `JOB_TITLE` | +| `ORDINALDIRECTION` | `LOCATION_OTHER` | +| `ORG` | `ORG` | +| `ORGANIZATION` | `ORGANIZATION` | +| `OTHER_NAME` | `ALIAS` | +| `PASS` | `PASSWORD` | +| `PASSPORT` | `PASSPORT` | +| `PASSPORTID` | `PASSPORT` | +| `PASSPORTNUM` | `PASSPORT` | +| `PASSWORD` | `PASSWORD` | +| `PATIENT` | `PATIENT_ID` | +| `PATIENT_ID` | `PATIENT_ID` | +| `PATIENT_NAME` | `NAME` | +| `PER` | `NAME` | +| `PERSON` | `PERSON` | +| `PHN` | `PHONE_NUMBER` | +| `PHONE` | `PHONE_NUMBER` | +| `PHONEIMEI` | `IMEI` | +| `PHONENUMBER` | `PHONE_NUMBER` | +| `PHONE_NUMBER` | `PHONE_NUMBER` | +| `PHOTO` | `FACE` | +| `PIN` | `PIN` | +| `POSTCODE` | `ADDRESS` | +| `PREFIX` | `PREFIX` | +| `PROFESSION` | `JOB_TITLE` | +| `PROVIDER` | `JOB_TITLE` | +| `PSP` | `PASSPORT` | +| `PWD` | `PASSWORD` | +| `ROUTING_NUMBER` | `FINANCIAL` | +| `RRN` | `NATIONAL_ID` | +| `SECADDRESS` | `ADDRESS` | +| `SECONDARYADDRESS` | `ADDRESS` | +| `SECURITYTOKEN` | `TOKEN` | +| `SEX` | `GENDER` | +| `SOCIALNUM` | `SSN` | +| `SOCIALNUMBER` | `SSN` | +| `SSN` | `SSN` | +| `STATE` | `ADDRESS` | +| `STREET` | `ADDRESS` | +| `STREETADDRESS` | `ADDRESS` | +| `SUFFIX` | `SUFFIX` | +| `SURNAME` | `NAME` | +| `SWIFT_CODE` | `FINANCIAL` | +| `TAXNUM` | `TAX_ID` | +| `TAX_NUMBER` | `NATIONAL_ID` | +| `TEL` | `PHONE_NUMBER` | +| `TELEPHONENUM` | `PHONE_NUMBER` | +| `TIME` | `TIME` | +| `TITLE` | `TITLE` | +| `URL` | `URL` | +| `USERAGENT` | `USER_AGENT` | +| `USERNAME` | `USERNAME` | +| `USER` | `USERNAME` | +| `US_BANK_NUMBER` | `NATIONAL_ID` | +| `US_DRIVER_LICENSE` | `DRIVER_LICENSE` | +| `US_ITIN` | `TAX_ID` | +| `US_LICENSE_PLATE` | `LICENSE_PLATE` | +| `US_PASSPORT` | `PASSPORT` | +| `US_SSN` | `SSN` | +| `VEHICLE` | `VEHICLE_ID` | +| `VEHICLEVIN` | `VIN` | +| `VEHICLEVRM` | `VEHICLE_ID` | +| `VIN` | `VIN` | +| `VRM` | `LICENSE_PLATE` | +| `ZIP` | `ADDRESS` | +| `ZIPCODE` | `ADDRESS` | +| `account_number` | `FINANCIAL` | +| `account_pin` | `FINANCIAL` | +| `api_key` | `API_KEY` | +| `audio_duration_range` | `NATIONAL_ID` | +| `audio_longer_than` | `NATIONAL_ID` | +| `audio_min_duration` | `DURATION` | +| `bank_routing_number` | `FINANCIAL` | +| `biometric_identifier` | `FACE` | +| `bitcoin_address` | `FINANCIAL` | +| `blood_type` | `BLOOD_TYPE` | +| `certificate_license_number` | `PROFESSIONAL_LICENSE` | +| `company_name` | `COMPANY` | +| `credit_card` | `FINANCIAL` | +| `credit_card_number` | `FINANCIAL` | +| `credit_card_security_code` | `FINANCIAL` | +| `credit_debit_card` | `FINANCIAL` | +| `customer_id` | `CUSTOMER_ID` | +| `date_of_birth` | `BIRTH_DATE` | +| `date_time` | `DATE_TIME` | +| `device_identifier` | `DEVICE_ID` | +| `driver_license_number` | `DRIVER_LICENSE` | +| `education_level` | `EDUCATION_LEVEL` | +| `employee_id` | `EMPLOYEE_ID` | +| `employment_status` | `EMPLOYMENT_STATUS` | +| `fax_number` | `FAX` | +| `first_name` | `NAME` | +| `health_plan_beneficiary_number` | `HEALTH_INSURANCE_ID` | +| `http_cookie` | `HTTP_COOKIE` | +| `ip_address` | `IP_ADDRESS` | +| `last_name` | `NAME` | +| `license_plate` | `LICENSE_PLATE` | +| `mac_address` | `MAC_ADDRESS` | +| `medical_record_number` | `MRN` | +| `phone_number` | `PHONE_NUMBER` | +| `political_view` | `POLITICAL_AFFILIATION` | +| `race_ethnicity` | `ETHNICITY` | +| `religious_belief` | `RELIGION` | +| `street_address` | `ADDRESS` | +| `swift_bic` | `FINANCIAL` | +| `swift_bic_code` | `FINANCIAL` | +| `tax_id` | `TAX_ID` | +| `unique_id` | `CUSTOMER_ID` | +| `user_name` | `USERNAME` | +| `vehicle_identifier` | `LICENSE_PLATE` | +| `AU_TAX_ID` | `TAX_ID` | +| `DE_TAX_ID` | `TAX_ID` | +| `CA_DRIVER_LICENSE` | `DRIVER_LICENSE` | +| `IN_DRIVER_LICENSE` | `DRIVER_LICENSE` | +| `UK_SSN` | `SSN` | +| `SG_SSN` | `SSN` | +| `FR_UNKNOWN_ENTITY` | `NATIONAL_ID` | +| `ES_UNKNOWN_ENTITY` | `NATIONAL_ID` | +| `AUSTRIA_PASSPORT_NUMBER` | `PASSPORT` | +| `HAITI_TAX_ID` | `TAX_ID` | +| `GERMANY_AAABBB` | `NATIONAL_ID` | +| `JAPAN_VEHICLE_NUMBER` | `NATIONAL_ID` | +| `NIGERIAN_NATIONAL_ID` | `NATIONAL_ID` | +| `FRENCH_PASSPORT` | `PASSPORT` | ## Proposed hierarchical entity mapping dictionary All entities are mapped to the 3rd level (canonical) From c74e4187288fa1acf367956d2ebbb84f4b40abe4 Mon Sep 17 00:00:00 2001 From: Omri Mendels Date: Tue, 24 Mar 2026 10:58:51 +0200 Subject: [PATCH 17/22] Enhance ADR-001 with additional evaluation details Added details about EvaluationResult and Error Analysis. --- docs/adr/ADR-001-simplified-evaluation-pipeline.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/adr/ADR-001-simplified-evaluation-pipeline.md b/docs/adr/ADR-001-simplified-evaluation-pipeline.md index 706e341e..82e75de9 100644 --- a/docs/adr/ADR-001-simplified-evaluation-pipeline.md +++ b/docs/adr/ADR-001-simplified-evaluation-pipeline.md @@ -18,7 +18,8 @@ Dataset (List[InputSample]) → evaluator.evaluate_all(dataset) # runs inference AND builds per-sample EvaluationResult objects → List[EvaluationResult] # intermediate per-sample carriers → evaluator.calculate_score(...) # calls get_results_dataframe() → DataFrame → scoring - → EvaluationResult (aggregated) + → EvaluationResult (aggregated) # contains both the PII level and the canonical level values + → Error Analysis ``` This design has four concrete pain points: From 5fdf5007dd36caeb32b249352887cb44beb275b8 Mon Sep 17 00:00:00 2001 From: Omri Mendels Date: Tue, 24 Mar 2026 11:05:11 +0200 Subject: [PATCH 18/22] Enhance evaluation pipeline with hierarchy scoring Refactor evaluation pipeline to include entity mapping and scoring per hierarchy. --- docs/adr/ADR-001-simplified-evaluation-pipeline.md | 13 ++++++++----- 1 file changed, 8 insertions(+), 5 deletions(-) diff --git a/docs/adr/ADR-001-simplified-evaluation-pipeline.md b/docs/adr/ADR-001-simplified-evaluation-pipeline.md index 82e75de9..830fadf8 100644 --- a/docs/adr/ADR-001-simplified-evaluation-pipeline.md +++ b/docs/adr/ADR-001-simplified-evaluation-pipeline.md @@ -60,14 +60,17 @@ results_df = model.predict_dataset(dataset) # NEW: returns the DataFrame direct # 3. Map entities (transforms both predictions and annotations into canonical entities) # ENTITY_MAPPING can be defined in config or as a shared constant. -results_df = map_entities(results_df, mapping=ENTITY_MAPPING) +mapper = CanonicalMapper() -# 4. Score -evaluator = SpanEvaluator() # no model argument needed! -results = evaluator.calculate_score_on_df(per_type=True, results_df=results_df) +# 4. Map to hierarchy (PII, High level, canonical, specific) and evaluate +evaluator = SpanEvaluator() +results_per_hierarchy = [] +for hierarchy in [1,2,3]): + results_df_hierarchy = mapper.map_entities(results_df, hierarchy=hierarchy) + results_per_hierarchy = evaluator.calculate_score_on_df(per_type=True, results_df=results_df) # 5. Analyze/plot -plotter = Plotter(results=results) +plotter = Plotter(results=results_per_hierarchy[0]) plotter.plot_scores() ``` From b400004cbe5f52c2497e124d41c8a0afa5976e3f Mon Sep 17 00:00:00 2001 From: Omri Mendels Date: Tue, 24 Mar 2026 11:07:16 +0200 Subject: [PATCH 19/22] Update ADR-001-simplified-evaluation-pipeline.md --- docs/adr/ADR-001-simplified-evaluation-pipeline.md | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-) diff --git a/docs/adr/ADR-001-simplified-evaluation-pipeline.md b/docs/adr/ADR-001-simplified-evaluation-pipeline.md index 830fadf8..0176973d 100644 --- a/docs/adr/ADR-001-simplified-evaluation-pipeline.md +++ b/docs/adr/ADR-001-simplified-evaluation-pipeline.md @@ -64,13 +64,12 @@ mapper = CanonicalMapper() # 4. Map to hierarchy (PII, High level, canonical, specific) and evaluate evaluator = SpanEvaluator() -results_per_hierarchy = [] -for hierarchy in [1,2,3]): - results_df_hierarchy = mapper.map_entities(results_df, hierarchy=hierarchy) - results_per_hierarchy = evaluator.calculate_score_on_df(per_type=True, results_df=results_df) + +results_df_mapped = mapper.map_entities(results_df) +results = evaluator.calculate_score_on_df(results_df=results_df_mapped) # 5. Analyze/plot -plotter = Plotter(results=results_per_hierarchy[0]) +plotter = Plotter(results=results) plotter.plot_scores() ``` From 7c600b6aaf9457a67ecb372c6a3a5e28dbf0ea1b Mon Sep 17 00:00:00 2001 From: Omri Mendels Date: Tue, 24 Mar 2026 11:08:05 +0200 Subject: [PATCH 20/22] Modify prediction method to return DataFrame Updated the prediction method to return a DataFrame directly. --- docs/adr/ADR-001-simplified-evaluation-pipeline.md | 1 - 1 file changed, 1 deletion(-) diff --git a/docs/adr/ADR-001-simplified-evaluation-pipeline.md b/docs/adr/ADR-001-simplified-evaluation-pipeline.md index 0176973d..8236da48 100644 --- a/docs/adr/ADR-001-simplified-evaluation-pipeline.md +++ b/docs/adr/ADR-001-simplified-evaluation-pipeline.md @@ -59,7 +59,6 @@ model = PresidioAnalyzerWrapper(analyzer_engine=AnalyzerEngine()) results_df = model.predict_dataset(dataset) # NEW: returns the DataFrame directly # 3. Map entities (transforms both predictions and annotations into canonical entities) -# ENTITY_MAPPING can be defined in config or as a shared constant. mapper = CanonicalMapper() # 4. Map to hierarchy (PII, High level, canonical, specific) and evaluate From 191f3304995592427316d645cbbae472e19a6c82 Mon Sep 17 00:00:00 2001 From: Omri Mendels Date: Tue, 24 Mar 2026 11:18:52 +0200 Subject: [PATCH 21/22] Enhance ADR-002 with input format and usage updates Added input format details for token comparisons and updated usage examples for CanonicalMapper. --- docs/adr/ADR-002-entity-mapping.md | 30 +++++++++++++++++++++++------- 1 file changed, 23 insertions(+), 7 deletions(-) diff --git a/docs/adr/ADR-002-entity-mapping.md b/docs/adr/ADR-002-entity-mapping.md index d0f53200..4ba6ab05 100644 --- a/docs/adr/ADR-002-entity-mapping.md +++ b/docs/adr/ADR-002-entity-mapping.md @@ -61,23 +61,36 @@ all outputs. ### Workflow ``` -CanonicalMapper.from_dataset(dataset) / CanonicalMapper.from_model(label_extractor) +CanonicalMapper.from_results_data_frame(results_df) → auto-resolve pass (EXACT → COUNTRY → COUNTRY_FALLBACK → FUZZY) → mapper.get_mapping(mode='html') # inspect gaps as an HTML table → mapper.map({...}) # programmatic assignment for pending labels → mapper.get_mapping() # returns dict[str, str | None] ``` +### Input +The input is the per-token comparison of predictions and actuals. +A user can get this by running the typical flow in presidio-evaluator, +or generate this in any other way. + +Format: + +| Column | Type | Description | +|---|---|---| +| `sentence_id` | `int` | Index of the source sentence in the dataset | +| `token` | `str` | Token string | +| `annotation` | `str` | Ground-truth entity tag (from `InputSample.tags`) | +| `prediction` | `str` | Model-predicted entity tag | + +For mapping, only the `annotation` and `prediction` are used. +The mapper returns a new data frame with updated columns. + ### Typical usage ```python -# Construct the mapper from a dataset or a model. -# Both constructors extract the label vocabulary, run the auto-resolve pass, +# Construct the mapper from the model's output (results data frame) # and return a CanonicalMapper instance. -mapper = CanonicalMapper.from_dataset(dataset) # dataset: List[InputSample] | pd.DataFrame -# or -mapper = CanonicalMapper.from_model(label_extractor) # label_extractor: LabelExtractor - +mapper = CanonicalMapper.from_results_data_frame(results_df) # Inspect unmapped labels — renders an HTML table highlighting gaps mapper.get_mapping(mode='html') @@ -86,6 +99,9 @@ mapper.map({"GGE": "ORG", "CustID": "CLIENT_ID", "MY_CUSTOM_LABEL": None}) # Retrieve the final mapping dict once all labels are resolved mapping: Dict[str, str] = mapper.get_mapping() + +# Update data frame +results_df_mapped: pd.DataFrame = mapper.get_mapped_results_dataframe() ``` ### Logging From 6725db79a05c2c0c362dc7318b550d3923a48955 Mon Sep 17 00:00:00 2001 From: Omri Mendels Date: Tue, 24 Mar 2026 11:25:06 +0200 Subject: [PATCH 22/22] Enhance ADR-001 with multi-hierarchical evaluation example Added example code for multi-hierarchical evaluations in the ADR document. --- .../ADR-001-simplified-evaluation-pipeline.md | 24 +++++++++++++++++++ 1 file changed, 24 insertions(+) diff --git a/docs/adr/ADR-001-simplified-evaluation-pipeline.md b/docs/adr/ADR-001-simplified-evaluation-pipeline.md index 8236da48..530098be 100644 --- a/docs/adr/ADR-001-simplified-evaluation-pipeline.md +++ b/docs/adr/ADR-001-simplified-evaluation-pipeline.md @@ -72,6 +72,30 @@ plotter = Plotter(results=results) plotter.plot_scores() ``` +For multi-hierarchical evaluations (on different granularities): +```python +# 1. Load dataset +dataset = InputSample.read_dataset_json("data/dataset.json") + +# 2. Choose model and run predictions → get DataFrame directly +model = PresidioAnalyzerWrapper(analyzer_engine=AnalyzerEngine()) +results_df = model.predict_dataset(dataset) # NEW: returns the DataFrame directly + +# 3. Map entities (transforms both predictions and annotations into canonical entities) +mapper = CanonicalMapper() + +# 4. Map to hierarchy (PII, High level, canonical, specific) and evaluate +evaluator = SpanEvaluator() +results_per_hierarchy = [] +for hierarchy in [1,2,3]): + results_df_hierarchy = mapper.map_entities(results_df, hierarchy=hierarchy) + results_per_hierarchy = evaluator.calculate_score_on_df(results_df=results_df_hierarchy) + +# 5. Analyze/plot +plotter = Plotter(results=results_per_hierarchy[0]) +plotter.plot_scores() +``` + ### Concrete changes | Component | Current | Proposed |