Skip to content

Add medtech model evidence export to MLflow - #47

Open
wyli wants to merge 5 commits into
devfrom
agent/mlflow-evidence-export
Open

Add medtech model evidence export to MLflow#47
wyli wants to merge 5 commits into
devfrom
agent/mlflow-evidence-export

Conversation

@wyli

@wyli wyli commented Aug 19, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • Add medtech-model-evidence-export as a post-hoc MLflow wrapper for medtech model-inference result JSON and evidence packs.
  • Use a domain-specific catalog name suitable for later synchronization to NVIDIA/skills, instead of publishing a generic mlflow-* utility.
  • Keep live training tracking in model-specific finetuning skills; this PR does not modify the training path from Add MLflow tracking and harden NV-Segment-CT fine-tuning #32.
  • Default to a no-network dry run and sanitized metadata export.

Review feedback addressed

  • Richer artifacts: log a four-file sanitized metadata bundle; optionally render PNG slice previews, attach existing visual summaries, or upload discovered NIfTI images/masks under explicit preview and all policies.
  • Parameters: log generation inputs, resolution/spacing, tumor conditioning, model identity, checkpoint names/digests, and rendered configuration values.
  • Quality metrics: export existing metrics and derive bounded-memory CT HU or generic intensity min/max/mean/std, an intensity-SNR heuristic, mask foreground percentage, and mapped tumor volume percentage.
  • Reproducibility seed: log the detected or caller-supplied seed as both a parameter and tag; tag missing seeds explicitly.
  • Prompt/config provenance: log recipe/config digests, prompt digests, repository revision, and an optional caller-supplied source reference.
  • Run notes: set mlflow.note.content to a concise human-readable summary.

Safety and scope

  • dry-run and metadata-only export are the defaults.
  • Live preview or all export requires --confirm-medical-artifact-upload.
  • Raw artifacts are size-capped and never included in the committed fixtures.
  • Likely secrets and common patient/study identifiers are excluded from logged parameters and metadata.
  • Declared permissions cover only the file, environment, network, and shell capabilities required by the exporter.
  • This is post-hoc inference export, not a model evaluator, live training tracker, model registry path, or clinical tool.

Validation

  • Focused exporter tests: 8 passed.
  • Repository tests: 429 passed.
  • Exporter evidence smoke: schema, sanity, runtime, cost, environment pin, and integrity gates passed.
  • Skill completeness: 20/20 Tier 1 and 29/29 Tier 2, with no advisories.
  • Full skill audit: 22/22 real targets passed.
  • Reproducibility audit: 22/22 targets passed.
  • Negative fixtures: 2/2 fired the expected gate.
  • Canonical evidence replay: no gate or payload drift.
  • Repository lint logic: zero warnings and zero errors.
  • Changed-file Ruff, isort, Black, YAML, Markdown, and diff-hygiene checks passed.
  • NVSkills CI: Tier 1, Tier 2, Tier 3, SkillCritic, and the content gate passed.
  • NVSkills benchmark verdict: PASS — recommended for publication; generated benchmark, skill card, and OMS signature attached by the signing bot.
  • Signed head: compliance, repository lint/test, required NVSkills status, and upstream signature verification passed.

A live remote MLflow or Databricks export was not run because it requires caller credentials and an approved artifact destination.

Related: #32 contains the separate training-loop MLflow integration.

AI-assisted: Created with Codex/GPT at the user's request.

Co-authored-by: Codex <noreply@openai.com>
Signed-off-by: Wenqi Li <wenqil@nvidia.com>
@wyli

wyli commented Aug 19, 2026

Copy link
Copy Markdown
Collaborator Author

/nvskills-ci

Co-authored-by: Codex <noreply@openai.com>

Signed-off-by: Wenqi Li <wenqil@nvidia.com>
@wyli

wyli commented Aug 19, 2026

Copy link
Copy Markdown
Collaborator Author

/nvskills-ci

Co-authored-by: Codex <noreply@openai.com>

Signed-off-by: Wenqi Li <wenqil@nvidia.com>
@wyli wyli changed the title Add post-hoc MLflow evidence export Add medtech model evidence export to MLflow Aug 19, 2026
@wyli

wyli commented Aug 19, 2026

Copy link
Copy Markdown
Collaborator Author

/nvskills-ci

Co-authored-by: Codex <noreply@openai.com>

Signed-off-by: Wenqi Li <wenqil@nvidia.com>
@wyli

wyli commented Aug 19, 2026

Copy link
Copy Markdown
Collaborator Author

/nvskills-ci

Signed-off-by: nvskills-svc-account <svc-nvskills-signing@nvidia.com>
@wyli
wyli marked this pull request as ready for review August 19, 2026 11:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants