Add optional MuRIL and IndicBERT encoder backbones

docs/indic-encoders.md

@@ -0,0 +1,92 @@
+# Indic encoder backbones
+
+OpenMed can expose a user-supplied MuRIL or IndicBERT backbone to an Indic
+entity adapter. This is an opt-in embedding contract, not a replacement for the
+complete Hindi and Telugu PII predictors used by `extract_pii` and
+`deidentify`.
+
+OpenMed does not bundle encoder weights and does not resolve a repository until
+you configure one. Listing PII models never downloads weights. A missing
+configuration, optional dependency, or checkpoint returns a skip reason so
+offline and base-package imports remain deterministic.
+
+## Supported families
+
+| Family | License | Capability |
+|---|---|---|
+| MuRIL | Apache-2.0 | 17 Indian languages plus transliterated counterparts |
+| IndicBERT | MIT | 12 languages |
+
+MuRIL was trained with transliterated segment pairs. It is therefore the
+preferred backbone for Hinglish and other code-mixed paths where an Indian
+language is written in Latin script. A backbone only produces aligned hidden
+states; an application still needs an entity adapter or task head evaluated on
+its own synthetic and governed data.
+
+## Configure and load
+
+Install the optional runtime without changing the base package footprint:
+
+```bash
+pip install "openmed[hf]" torch
+```
+
+Then configure either a repository identifier that you have chosen or an
+existing local directory:
+
+```python
+from openmed.core.model_registry import (
+    configure_indic_encoder,
+    get_pii_models_by_language,
+    load_configured_indic_encoder,
+)
+
+configure_indic_encoder(
+    "google/muril-base-cased",
+    family="muril",
+    languages=("hi", "te"),
+    local_files_only=True,
+)
+
+# Existing dedicated predictors remain present; the configured backbone is
+# added only when its optional runtime dependencies are importable.
+models = get_pii_models_by_language("hi")
+
+resolution = load_configured_indic_encoder()
+if not resolution.available:
+    print(resolution.skip_reason)
+else:
+    encoded = resolution.handle.encode("Patient Asha Verma ka record dekhiye.")
+    print(encoded.last_hidden_state.shape, encoded.offset_mapping)
+```
+
+Set `local_files_only=False` only when you intentionally want the explicitly
+configured repository to be resolved through the model host. Private or gated
+repositories can use the `token=` argument. Tokens are held only in the
+process-local configuration and are excluded from its representation.
+
+The encoder output contains tokenizer tensors, an attention mask, character
+offsets, final hidden states, and a SHA-256 input identifier. It contains no raw
+input text. Encoding diagnostics log only that digest and aggregate lengths.
+
+Use `clear_indic_encoder_config()` to remove the process-local configuration.
+
+## Model-card provenance
+
+When publishing metadata for an adapter that consumes one of these backbones,
+add an `encoder_provenance` object to the manifest row passed to the model-card
+renderer:
+
+```json
+{
+  "family": "MuRIL",
+  "source": "google/muril-base-cased",
+  "license": "Apache-2.0",
+  "provenance": "user-supplied",
+  "weights": "user-supplied; not bundled",
+  "supports_transliterated_text": true
+}
+```
+
+The rendered model card includes these values in a dedicated Encoder
+Provenance section.

mkdocs.yml

@@ -33,6 +33,7 @@ nav:
       - ModelLoader & Pipelines: model-loader.md
       - Model Registry: model-registry.md
       - Model Manifest: model-manifest.md
+      - Indic Encoder Backbones: indic-encoders.md
       - Advanced NER & Output Formatting: output-formatting.md
       - Clinical Context & Extraction Depth: clinical/context-and-extraction.md
       - Medication Sig Parser: clinical/sig-parser.md

openmed/core/model_card.py

@@ -215,6 +215,9 @@ def render_model_card(row: dict[str, Any]) -> str:
     training_provenance_lines = _training_provenance_block(row)
     if training_provenance_lines:
         lines.extend(["", "## Training Provenance", "", *training_provenance_lines])
+    encoder_provenance_lines = _encoder_provenance_block(row)
+    if encoder_provenance_lines:
+        lines.extend(["", "## Encoder Provenance", "", *encoder_provenance_lines])
     return "\n".join(lines) + "\n"
 
 
@@ -769,6 +772,41 @@ def _training_provenance_block(row: dict[str, Any]) -> list[str]:
     return lines
 
 
+def _encoder_provenance_block(row: dict[str, Any]) -> list[str]:
+    payload = row.get("encoder_provenance")
+    if not isinstance(payload, dict):
+        payload = row.get("indic_encoder")
+    if not isinstance(payload, dict):
+        return []
+
+    family = _string(payload.get("family"), "Not specified")
+    source = _string(
+        payload.get("source") or payload.get("repo_id"),
+        "Not specified",
+    )
+    license_name = _string(payload.get("license"), "Not specified")
+    provenance = _string(payload.get("provenance"), "user-supplied")
+    weights = _string(payload.get("weights"), "user-supplied; not bundled")
+    transliterated = payload.get("supports_transliterated_text")
+    transliterated_value = (
+        "Yes"
+        if transliterated is True
+        else "No"
+        if transliterated is False
+        else "Not specified"
+    )
+    return [
+        "| Field | Value |",
+        "|---|---|",
+        f"| Encoder family | {family} |",
+        f"| Source | `{source}` |",
+        f"| License | {license_name} |",
+        f"| Provenance | {provenance} |",
+        f"| Weights | {weights} |",
+        f"| Transliterated text | {transliterated_value} |",
+    ]
+
+
 def _gate_status(value: Any) -> str:
     if value is True:
         return "passed"