feat: add SMS short-text de-identification

docs/examples.md

@@ -70,6 +70,7 @@ content-security policy blocks the optional inline styling.
 | --- | --- |
 | `examples/pii_model_comparison.py` | Compares multiple PII models across shared sample text and summarizes extraction quality. |
 | `examples/pii_batch_processing.py` | Runs batch PII extraction and de-identification with `BatchProcessor(operation=...)`. |
+| `examples/sms_deid_helpdesk_logs.py` | Redacts RapidPro-style JSON or generic CSV helpdesk exports with the `short_text` preset, contact pseudonymization, timestamp coarsening, and bounded batches. |
 | `examples/pii_multilingual_new_languages.py` | Exercises Dutch, Hindi, Telugu, Portuguese, Arabic, Japanese, and Turkish registry entries, locale-specific regex matches, and optional live extraction with the new public checkpoints. |
 | `examples/gradio_deid_app.py` | Interactive Gradio UI to paste synthetic text, pick a `mask`/`replace`/`hash` method, and view the de-identified output plus detected entities (optional `pip install gradio`). |
 | `examples/v16_policy_audit_release_gates.py` | Demonstrates v1.6 policy profiles, canonical spans, signed audit reports, review bundles, redaction previews, leakage heatmaps, and k-anonymity metrics without model downloads. |

docs/sms-short-text-deid.md

@@ -0,0 +1,163 @@
+# SMS short-text de-identification
+
+OpenMed's `short_text` preset is for clinical messages up to SMS scale,
+including MomConnect-style helpdesk conversations and RapidPro flow/message
+exports. It runs locally and combines the normal PII pipeline with deterministic
+rules for message formats where sentence context is sparse or absent.
+
+RapidPro represents SMS addresses as contact URNs such as
+`tel:+250788123123`, and a logical message can span more than one physical SMS.
+The adapter therefore treats the message record—not a sentence—as the unit of
+work. See RapidPro's documentation for its
+[contact URN](https://rapidpro.github.io/rapidpro/docs/contacts/) and
+[message](https://rapidpro.github.io/rapidpro/docs/msgs/) models.
+
+## What the preset changes
+
+The preset is named `short_text` and uses a lower high-recall threshold than
+the default clinical-note path. It adds context-independent recognition for:
+
+- Kenyan, Ugandan, and South African E.164-style MSISDNs (`+254`, `+256`,
+  `+27`), spaced or punctuated variants, local `07xx` numbers, and 4–6 digit
+  shortcodes;
+- contextual and common-shape national/medical identifiers;
+- names following code-mixed maternal-health honorifics such as `mama`,
+  `maama`, `sisi`, `dada`, `mme`, and `ndugu`, including all-caps messages;
+- common SMS and clinical abbreviations (`ANC`, `EDD`, `LMP`, `CHW`, `pls`,
+  `appt`, and similar) as allowed terms so a sparse context window does not turn
+  them into identifiers.
+
+Detection may normalize whitespace internally, but replacement offsets are
+mapped to the original string. The preset never truncates, wraps, or otherwise
+reflows source text.
+
+## One message
+
+Install the model-backed PII dependencies once, then keep the model cached for
+offline use:
+
+```bash
+uv pip install -e ".[hf]"
+```
+
+```python
+from openmed.multimodal import deidentify_short_text
+
+message = "  HABARI MAMA Amina Njeri call +254712345678 ANC  "
+result = deidentify_short_text(message)
+
+assert result.original_text == message
+print(result.deidentified_text)
+# "  HABARI [PERSON] call [PHONE] ANC  "
+```
+
+`deidentify_short_text` defaults to mask replacements and accepts the normal
+OpenMed model, language, policy, loader, and mapping options. Code-mixed rules
+remain language-independent; `lang` is still passed to the model-backed and
+standard locale recognizers.
+
+## RapidPro-shaped JSON
+
+The JSON adapter accepts a top-level list, a single message record, or an object
+whose record list is under `messages`, `results`, `runs`, `records`, or nested
+`data`. Every string field named `text`, including flow-result `input.text`, is
+de-identified. The surrounding object, row order, flow UUIDs, directions, and
+other non-sensitive fields are retained.
+
+```python
+import os
+
+from openmed.multimodal import redact_sms_json
+
+result = redact_sms_json(
+    "rapidpro-message-export.json",
+    "rapidpro-message-export.redacted.json",
+    batch_size=512,
+)
+print(result.summary.to_dict())
+```
+
+Contact `urn`, `urns`, `contact_urn`, and string-valued `contact` fields are
+replaced with HMAC-SHA256 pseudonyms. Nested contact `name`, `urn`, and
+`address` values are pseudonymized too; contact UUIDs and flow UUIDs remain
+intact. Repeated source values receive the same pseudonym within one run.
+
+By default a fresh random HMAC key is generated per run, so outputs from
+different runs cannot be linked. Supply a secret key only when an authorized
+workflow needs stable cross-run linkage:
+
+```python
+result = redact_sms_json(
+    "messages.json",
+    "messages.redacted.json",
+    contact_hash_key=os.environ["OPENMED_SMS_CONTACT_HASH_KEY"],
+)
+```
+
+Keep that key outside source control and logs. The summary contains only counts,
+never contact values or message text.
+
+## Generic CSV and high-volume logs
+
+Generic CSV input must include `text` and at least one of `urn` or `contact`.
+The expected RapidPro-compatible core columns are:
+
+```text
+urn,contact,direction,text,sent_on
+```
+
+Extra columns are preserved in their original order. When an output destination
+is supplied, the CSV path reads and writes incrementally, while
+`iter_redacted_sms_records` accepts any iterable of mapping records. Both keep
+constant-size contact-hashing state, retain at most one source batch in memory,
+and reuse OpenMed's `BatchProcessor` for batched model inference. Omitting the
+CSV output destination materializes the result text for convenience.
+
+```python
+from openmed.multimodal import redact_sms_csv
+
+result = redact_sms_csv(
+    "helpdesk.csv",
+    "helpdesk.redacted.csv",
+    batch_size=512,
+)
+assert result.summary.row_count > 0
+```
+
+For database cursors or other record streams:
+
+```python
+from openmed.multimodal import iter_redacted_sms_records
+
+for redacted_record in iter_redacted_sms_records(database_cursor, batch_size=512):
+    write_record(redacted_record)
+```
+
+The outer record batch may be larger than 100 rows; the existing batch utility
+automatically uses its validated inference chunk size inside that bound.
+
+## Timestamp handling and verification
+
+ISO-like values in `sent_on`, `received_on`, `delivered_on`, `created_on`,
+`modified_on`, and `timestamp` are coarsened to `YYYY-MM-DD`. Invalid or custom
+timestamp strings are retained rather than guessed.
+
+Before an export crosses a trust boundary, verify known synthetic or
+run-scoped source identifiers with `assert_redacted`:
+
+```python
+from openmed.interop import assert_redacted
+
+assert_redacted(redacted_text, replacement_to_original_mapping)
+```
+
+Use only synthetic fixtures in tests and benchmarks. Do not log raw input rows,
+model entities, or failed records; the adapter's error and summary paths are
+designed to report counts and error types without message content.
+
+## Boundaries
+
+This adapter does not call a live RapidPro API, serve webhooks, classify message
+intent, or process WhatsApp media. It handles exported text payloads only. JSON
+documents are materialized to preserve arbitrary wrapper schemas; for exports
+with millions of rows, prefer streaming CSV or `iter_redacted_sms_records`.

examples/sms_deid_helpdesk_logs.py

@@ -0,0 +1,44 @@
+#!/usr/bin/env python3
+"""De-identify a RapidPro-style JSON or generic CSV message export."""
+
+from __future__ import annotations
+
+import argparse
+import json
+import os
+from pathlib import Path
+
+from openmed.multimodal import redact_sms_export
+
+HASH_KEY_ENV = "OPENMED_SMS_CONTACT_HASH_KEY"
+
+
+def build_parser() -> argparse.ArgumentParser:
+    parser = argparse.ArgumentParser(
+        description="Redact an SMS helpdesk export with the short_text preset."
+    )
+    parser.add_argument("input", type=Path, help="Source .json or .csv export")
+    parser.add_argument("output", type=Path, help="Destination redacted export")
+    parser.add_argument(
+        "--batch-size",
+        type=int,
+        default=512,
+        help="Maximum source records retained per redaction batch",
+    )
+    return parser
+
+
+def main() -> int:
+    args = build_parser().parse_args()
+    result = redact_sms_export(
+        args.input,
+        args.output,
+        batch_size=args.batch_size,
+        contact_hash_key=os.environ.get(HASH_KEY_ENV),
+    )
+    print(json.dumps(result.summary.to_dict(), sort_keys=True))
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

mkdocs.yml

@@ -61,6 +61,7 @@ nav:
       - GPTQ Export: export-gptq.md
   - PII & De-identification:
       - De-identification API: api/deidentification.md
+      - SMS Short-Text De-identification: sms-short-text-deid.md
       - PII Anonymization: anonymization.md
       - Per-Language De-identification: languages.md
       - Smart Entity Merging: pii-smart-merging.md

openmed/core/pii.py

@@ -606,15 +606,16 @@ def _prepare_pii_text(
     lang: str,
     normalize_accents: Optional[bool],
     config: Optional[OpenMedConfig] = None,
+    preserve_whitespace: bool = False,
 ) -> _PreparedPIIText:
-    """Return original stripped text, inference text, and normalization flag."""
+    """Return source text, inference text, and normalization metadata."""
     do_normalize = (
         normalize_accents
         if normalize_accents is not None
         else (lang in _ACCENT_NORMALIZE_LANGS)
     )
 
-    original_text = text.strip()
+    original_text = text if preserve_whitespace else text.strip()
     width_convention = config.cjk_width_convention if config is not None else "cjk"
     detection_normalization = normalize_for_pii_detection(
         original_text,
@@ -798,6 +799,7 @@ def _extract_pii_batch(
     custom_recognizer: Any = None,
     abdm: Optional[bool] = None,
     *,
+    preserve_whitespace: bool = False,
     locale: Optional[str] = None,
     loader: Optional["ModelLoader"] = None,
     privacy_filter_pipeline: Optional[Any] = None,
@@ -811,6 +813,7 @@ def _extract_pii_batch(
             lang=lang,
             normalize_accents=normalize_accents,
             config=config,
+            preserve_whitespace=preserve_whitespace,
         )
         for text in texts
     ]
@@ -918,6 +921,7 @@ def extract_pii(
     max_cache_entries: int = 128,
     normalize_accents: Optional[bool] = None,
     *,
+    preserve_whitespace: bool = False,
     locale: Optional[str] = None,
     loader: Optional["ModelLoader"] = None,
     batch_size: Optional[int] = None,
@@ -951,6 +955,8 @@ def extract_pii(
             names.  Entity spans in the result reference the *original*
             (accented) text.  ``None`` (default) auto-enables for languages
             in ``_ACCENT_NORMALIZE_LANGS`` (currently Spanish).
+        preserve_whitespace: Preserve leading and trailing source whitespace
+            so returned entity offsets refer to the exact input string.
         loader: Optional shared model loader to reuse warmed pipelines.
         batch_size: Optional backend inference batch size.
         num_workers: Optional backend inference worker count.
@@ -1015,6 +1021,7 @@ def extract_pii(
         use_smart_merging=use_smart_merging,
         lang=lang,
         normalize_accents=normalize_accents,
+        preserve_whitespace=preserve_whitespace,
         locale=locale,
         loader=loader,
         custom_recognizer=custom_recognizer,
@@ -1826,6 +1833,7 @@ def _deidentify_batch(
     lang: str = "en",
     normalize_accents: Optional[bool] = None,
     use_safety_sweep: bool = True,
+    preserve_whitespace: bool = False,
     custom_recognizer: Any = None,
     abdm: Optional[bool] = None,
     policy: Optional[str] = None,
@@ -1847,16 +1855,17 @@ def _deidentify_batch(
         date_shift_max_days=date_shift_max_days,
         date_shift_secret=date_shift_secret,
     )
-    stripped_texts = [text.strip() for text in texts]
+    source_texts = [text if preserve_whitespace else text.strip() for text in texts]
     recognizer = coerce_custom_recognizer(custom_recognizer)
     pii_results = _extract_pii_batch(
-        stripped_texts,
+        source_texts,
         model_name=model_name,
         confidence_threshold=confidence_threshold,
         config=config,
         use_smart_merging=use_smart_merging,
         lang=lang,
         normalize_accents=normalize_accents,
+        preserve_whitespace=preserve_whitespace,
         locale=locale,
         custom_recognizer=recognizer,
         abdm=abdm_mode_enabled(
@@ -1872,14 +1881,14 @@ def _deidentify_batch(
 
     if use_safety_sweep:
         swept_results = []
-        for stripped_text, pii_result in zip(stripped_texts, pii_results):
+        for source_text, pii_result in zip(source_texts, pii_results):
             pii_result, _ = _apply_safety_sweep_to_result(
-                stripped_text,
+                source_text,
                 pii_result,
                 lang=lang,
                 locale=locale,
             )
-            _suppress_custom_allowed_entities(stripped_text, pii_result, recognizer)
+            _suppress_custom_allowed_entities(source_text, pii_result, recognizer)
             swept_results.append(pii_result)
         pii_results = swept_results
 
@@ -1906,7 +1915,7 @@ def _deidentify_batch(
             surrogate_vault=surrogate_vault,
             policy=policy or "hipaa_safe_harbor",
         )
-        for text, pii_result in zip(stripped_texts, pii_results)
+        for text, pii_result in zip(source_texts, pii_results)
     ]
 
 

openmed/core/pipeline.py

@@ -243,6 +243,7 @@ def __init__(
         lang: str = "en",
         normalize_accents: Optional[bool] = None,
         use_safety_sweep: bool = True,
+        preserve_whitespace: bool = False,
         loader: Any = None,
         privacy_filter_pipeline: Any = None,
         model_detector: ModelDetector | None = None,
@@ -292,6 +293,7 @@ def __init__(
         self.lang = lang
         self.normalize_accents = normalize_accents
         self.use_safety_sweep = use_safety_sweep
+        self.preserve_whitespace = bool(preserve_whitespace)
         self.loader = loader
         self.privacy_filter_pipeline = privacy_filter_pipeline
         self.model_detector = model_detector
@@ -354,7 +356,7 @@ def run(
             stage_name: 0.0 for stage_name in STAGE_NAMES
         }
         cascade_duration_ms: float | None = None
-        original_text = text.strip()
+        original_text = text if self.preserve_whitespace else text.strip()
         with _stage_timer(stage_durations_ms, STAGE_NAMES[0]):
             normalized = self.stage1_normalize(original_text)
         with _stage_timer(stage_durations_ms, STAGE_NAMES[1]):
@@ -960,6 +962,7 @@ def stage5_fast_pii_model(self, text: str, route: LanguageRoute) -> Any:
             self.use_smart_merging,
             lang=route.lang,
             normalize_accents=self.normalize_accents,
+            preserve_whitespace=self.preserve_whitespace,
             loader=self.loader,
             batch_size=getattr(self.config, "batch_size", None),
             num_workers=getattr(self.config, "num_workers", None),