Skip to content

Add a dataflow-tool processor for in-flow record redaction#1773

Draft
maziyarpanahi wants to merge 1 commit into
masterfrom
feature/om-498-dataflow-tool-processor
Draft

Add a dataflow-tool processor for in-flow record redaction#1773
maziyarpanahi wants to merge 1 commit into
masterfrom
feature/om-498-dataflow-tool-processor

Conversation

@maziyarpanahi

Copy link
Copy Markdown
Owner

Pull Request

Description

Adds a scriptable in-flow record redaction adapter that preserves record cardinality and incoming attributes while emitting only numeric redaction counters. A module-level loader cache keeps the model pipeline resident across records handled by the same worker.

Type of Change

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Documentation update
  • Code refactoring
  • Performance improvement
  • Test addition/improvement

Changes Made

  • Add record and flow-file callables with dotted-field selection, fail-closed batch validation, and PHI-safe count attributes.
  • Add a persistent JSON-lines entrypoint and module-level, thread-safe pipeline reuse.
  • Document embedded and external processor wiring, offline operation, and attribute-safety boundaries.
  • Add offline tests for redaction, pass-through behavior, exact cardinality, cache reuse, and PHI-safe failures.

Testing

  • I have added tests that prove my feature works
  • New and existing unit tests pass locally with my changes
  • I have tested this change with different models/inputs

Validation completed:

  • .venv/bin/python -m pytest tests/unit/integrations/test_dataflow_tool_processor.py -q — 5 passed
  • .venv/bin/python -m pytest tests/ -q — 5203 passed, 46 skipped
  • make format
  • make lint
  • make format-check
  • make type-check
  • pre-commit run --files openmed/integrations/dataflow_tool_processor.py openmed/integrations/__init__.py tests/unit/integrations/test_dataflow_tool_processor.py examples/dataflow-tool/README.md

Documentation

  • I have updated the documentation accordingly
  • I have added docstrings to new functions/classes
  • I have updated the CHANGELOG.md

Code Quality

  • I ran make format, make lint, and make format-check
  • For Swift/OpenMedKit changes, I ran make format-swift and make lint-swift
  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • My changes generate no new warnings

Dependencies

  • I have not added any new dependencies
  • OR I have added new dependencies and they are justified because: ____

Checklist

  • I have read the contributing guidelines
  • My commits have clear, descriptive messages
  • I have squashed/organized my commits appropriately

Related Issues

Closes #855

Screenshots/Examples

The new example documents a synthetic record, the redacted output contract, and the numeric attributes produced for downstream routing.

@maziyarpanahi maziyarpanahi added help wanted Extra attention is needed good first issue Good for newcomers roadmap-v2 OpenMed V2 roadmap backlog feature New capability P3 Strategic labels Jul 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

feature New capability good first issue Good for newcomers help wanted Extra attention is needed P3 Strategic roadmap-v2 OpenMed V2 roadmap backlog

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add a dataflow-tool processor for in-flow record redaction

1 participant