Skip to content

feat: add SMS short-text de-identification#1841

Merged
maziyarpanahi merged 2 commits into
masterfrom
feature/om-852-feat-sms-short-text-mode
Jul 22, 2026
Merged

feat: add SMS short-text de-identification#1841
maziyarpanahi merged 2 commits into
masterfrom
feature/om-852-feat-sms-short-text-mode

Conversation

@maziyarpanahi

@maziyarpanahi maziyarpanahi commented Jul 20, 2026

Copy link
Copy Markdown
Owner

Description

Adds an SMS-scale short_text de-identification preset and export adapters for RapidPro-shaped JSON and generic CSV helpdesk logs. The path preserves message offsets and source schemas, pseudonymizes contact values, coarsens timestamps, and processes large record streams in bounded batches.

Type of Change

  • New feature (non-breaking change which adds functionality)
  • Documentation update
  • Performance improvement
  • Test addition/improvement

Changes Made

  • Add code-mixed, no-punctuation-tolerant rules for African phone numbers, shortcodes, IDs, honorific-led names, and SMS abbreviations.
  • Add JSON, CSV, and iterable record handlers with deterministic per-run contact hashing, timestamp coarsening, and batched de-identification.
  • Add a 600-message synthetic recall and no-leak benchmark, export round-trip and offset-fidelity tests, fixtures, an example, and user documentation.
  • Preserve leading and trailing source whitespace through the pipeline and batch paths without changing the default clinical-note behavior.

Testing

  • The 600-message corpus reaches complete seeded phone/name/ID redaction and higher recall than the default preset.
  • RapidPro JSON and generic CSV round trips preserve schemas, row counts, flow fields, and consistent contact pseudonyms.
  • The focused SMS suite passes: 6 passed.
  • The full repository suite passes on current master: 6,045 passed, 48 skipped.

Additional validation:

  • make format
  • make lint
  • make format-check
  • make type-check
  • pre-commit run --files <changed files>
  • make docs-build
  • python3 -m build

Documentation

  • Added the SMS short-text guide and examples navigation.
  • Added docstrings to new public functions and classes.
  • CHANGELOG.md is unchanged; release notes can include this feature when it ships.

Code Quality

  • Canonical formatting, lint, type, pre-commit, and strict documentation checks pass.
  • The complete diff was reviewed.
  • Error and summary paths avoid raw contact values and message text.
  • No new warnings are introduced.

Dependencies

  • No new dependencies were added.

Checklist

  • Read the contributing guidelines.
  • Used clear, focused commits.
  • Kept the branch scoped to OM-852.

Related Issues

Closes #1429.

Screenshots/Examples

See examples/sms_deid_helpdesk_logs.py and docs/sms-short-text-deid.md for runnable JSON and CSV usage.

@maziyarpanahi maziyarpanahi added roadmap-v2 OpenMed V2 roadmap backlog feature New capability P0 Critical labels Jul 20, 2026
@maziyarpanahi
maziyarpanahi force-pushed the feature/om-852-feat-sms-short-text-mode branch from 9484669 to 6a3317e Compare July 22, 2026 03:10
@maziyarpanahi

Copy link
Copy Markdown
Owner Author

Ready to merge. I rebased both commits onto current master and verified the complete OM-852 acceptance surface: the 600-message synthetic corpus has complete seeded phone/name/ID redaction and higher recall than the default preset; RapidPro JSON and generic CSV round trips preserve schema and flow fields; contact pseudonyms are stable within a run; batching stays bounded; and whitespace/offset fidelity is preserved without changing the default clinical-note path.

Validation is complete: the focused SMS suite passes, the full local suite passes (6,045 passed, 48 skipped), formatting/lint/type checks, all changed-file pre-commit hooks, strict docs, and package build pass. Hosted CI is fully green across Linux, macOS, Windows, security/SBOM/policy checks, and amd64/arm64 container smoke tests; the one macOS timing flake cleared on rerun without code changes. The diff and PR metadata are clean of prohibited vendor/assistant names and co-author trailers. This closes #1429.

@maziyarpanahi
maziyarpanahi merged commit 02eca8d into master Jul 22, 2026
29 of 34 checks passed
@maziyarpanahi
maziyarpanahi deleted the feature/om-852-feat-sms-short-text-mode branch July 22, 2026 03:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

feature New capability P0 Critical roadmap-v2 OpenMed V2 roadmap backlog

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Short-message clinical de-identification mode for SMS-scale text

1 participant