Skip to content

Add conservative Indic morphology boundary refinement#1764

Open
maziyarpanahi wants to merge 1 commit into
masterfrom
feature/om-661-indic-morphology-sandhi-boundary-refine
Open

Add conservative Indic morphology boundary refinement#1764
maziyarpanahi wants to merge 1 commit into
masterfrom
feature/om-661-indic-morphology-sandhi-boundary-refine

Conversation

@maziyarpanahi

@maziyarpanahi maziyarpanahi commented Jul 19, 2026

Copy link
Copy Markdown
Owner

Description

Adds an opt-in, precision-first morphology layer for Hindi and Marathi Devanagari plus Tamil, Telugu, Malayalam, and Kannada name spans. It conservatively strips frequent case-marker and postposition suffixes only when both the upstream confidence threshold and an exact caller-provided stem allow-list pass, preserves grapheme-aligned source offsets, and reverses supported Tamil glide sandhi with explicit source mappings. The existing span path remains unchanged unless the layer is enabled.

Type of Change

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Documentation update
  • Code refactoring
  • Performance improvement
  • Test addition/improvement

Changes Made

  • Added conservative cross-script suffix tables, stemming, sandhi splitting, idempotence, grapheme segmentation, and output-to-source offset maps.
  • Added an opt-in decoder boundary-refinement hook restricted to canonical person-name labels and guarded by confidence plus an exact stem allow-list.
  • Added a synthetic six-language fixture and held-out suffix-like names covering precision, offset reconstruction, over-stripping, idempotence, and disabled-path safety.

Testing

  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes
  • I have tested this change with different models/inputs
  • .venv/bin/python -m pytest tests/ -q — 5,206 passed, 55 skipped
  • make format, make lint, and make format-check
  • File-scoped pre-commit hooks, including Bandit and secret detection

Documentation

  • I have updated the documentation accordingly
  • I have added docstrings to new functions/classes
  • I have updated the CHANGELOG.md

No user-facing documentation or changelog entry is needed for this internal opt-in hook; the public APIs and offset contracts are documented in code.

Code Quality

  • I ran make format, make lint, and make format-check
  • For Swift/OpenMedKit changes, I ran make format-swift and make lint-swift
  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • My changes generate no new warnings

Swift/OpenMedKit validation is not applicable because this PR changes only Python code and synthetic test data.

Dependencies

  • I have not added any new dependencies
  • OR I have added new dependencies and they are justified because: ____

Checklist

  • I have read the contributing guidelines
  • My commits have clear, descriptive messages
  • I have squashed/organized my commits appropriately

Related Issues

Closes #1487

Screenshots/Examples

Not applicable; this change has no UI surface.

@maziyarpanahi maziyarpanahi added roadmap-v2 OpenMed V2 roadmap backlog feature New capability P2 Medium labels Jul 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

feature New capability P2 Medium roadmap-v2 OpenMed V2 roadmap backlog

Projects

None yet

1 participant