Skip to content

Add MLflow tracking and harden NV-Segment-CT fine-tuning - #32

Open
wyli wants to merge 10 commits into
devfrom
agent/mlflow-finetune-demo
Open

Add MLflow tracking and harden NV-Segment-CT fine-tuning#32
wyli wants to merge 10 commits into
devfrom
agent/mlflow-finetune-demo

Conversation

@wyli

@wyli wyli commented Jul 7, 2026

Copy link
Copy Markdown
Collaborator

Summary

Add opt-in MLflow tracking to nv-segment-ct-finetune while keeping the upstream MONAI bundle as the sole training implementation. The change also tightens runtime reproducibility, makes bundle staging explicit, and refreshes the skill's documentation and publication evidence.

What changed

  • Add mlflow_experiment_name, mlflow_tracking_uri, and mlflow_run_name inputs.
    • Setting an experiment name enables MONAI 1.4.0's built-in --tracking mlflow path.
    • Omitting the tracking URI uses a local file store under <output-dir>/mlruns.
    • Caller-configured remote stores, including Databricks, remain supported.
    • Non-secret tracking metadata is recorded in output.json; executed-config uploads are disabled.
  • Preserve the existing training behavior: no custom MLflow run lifecycle, trainer replacement, DataLoader override, or training-config patching.
  • Require the exact behavior-bearing monai==1.4.0 version and bound the optional MLflow dependency to >=2.10,<4.
  • Replace runtime downloads, environment creation, package installation, and fallback repair logic with one explicit local staging path.
  • Verify all seven pinned bundle assets by SHA-256 before MONAI starts, and load checkpoint comparisons with weights_only=True.
  • Document required permissions, network endpoints, filesystem writes, caller-managed credentials, setup steps, and failure behavior.
  • Bump the skill manifest to 0.4.2 and refresh the skill index, reference material, benchmark report, skill card, tests, and OMS signature.

Validation

  • make test — 424 passed.
  • make verify-skills — 21/21 real targets passed completeness and reproducibility audits with no advisories.
  • make verify-negative-fixtures — 2/2 fixtures failed at their expected gates.
  • make verify — repository lint, hooks, canonical pack replay, and drift checks passed.
  • Focused nv-segment-ct-finetune tests — 17 passed.
  • Pinned local staging and integrity checks passed for the declared configs, label dictionary, metadata, and model weights.
  • The refreshed NVSkills evaluation and signature verification passed.
  • Current GitHub lint/test, compliance, and NVSkills CI checks are green.

Scope and limitations

  • A live GPU fine-tuning run was not repeated for the final implementation-only simplification.
  • MLflow remains optional. Tracking failures propagate from the upstream MONAI run.
  • This skill is for development workflows and is not for clinical deployment, clinical interpretation, autonomous diagnosis, or regulatory submission.

AI-assisted: Created with Codex/GPT at the user's request.

Add a publishable evidence-export skill and optional MONAI-native live tracking for NV-Segment-CT finetuning. Pin the behavior-bearing Task06 trainer and exact MONAI runtime without changing the original training hyperparameters.

Co-authored-by: Codex <noreply@openai.com>
@wyli
wyli force-pushed the agent/mlflow-finetune-demo branch from c5a5879 to c063646 Compare July 8, 2026 12:22
wyli and others added 2 commits August 11, 2026 15:24
Use MONAI's built-in tracking path and remove the unrelated generic exporter prototype.

Co-authored-by: Codex <noreply@openai.com>
Signed-off-by: Wenqi Li <wenqil@nvidia.com>
@wyli

wyli commented Aug 17, 2026

Copy link
Copy Markdown
Collaborator Author

/nvskills-ci

Remove automatic runtime re-exec and verify behavior-bearing bundle assets before execution.

Co-authored-by: Codex <noreply@openai.com>
Signed-off-by: Wenqi Li <wenqil@nvidia.com>
@wyli

wyli commented Aug 17, 2026

Copy link
Copy Markdown
Collaborator Author

/nvskills-ci

Document first-run bundle downloads and optional remote MLflow access for NVSkills least-privilege validation.

Co-authored-by: Codex <noreply@openai.com>
Signed-off-by: Wenqi Li <wenqil@nvidia.com>
@wyli

wyli commented Aug 17, 2026

Copy link
Copy Markdown
Collaborator Author

/nvskills-ci

Remove runtime artifact downloads and require pinned local assets before execution.

Co-authored-by: Codex <noreply@openai.com>
Signed-off-by: Wenqi Li <wenqil@nvidia.com>
@wyli

wyli commented Aug 17, 2026

Copy link
Copy Markdown
Collaborator Author

/nvskills-ci

List the script capabilities detected by SkillSpector without broadening the agent tool allowance.

Co-authored-by: Codex <noreply@openai.com>
Signed-off-by: Wenqi Li <wenqil@nvidia.com>
@wyli

wyli commented Aug 17, 2026

Copy link
Copy Markdown
Collaborator Author

/nvskills-ci

svc-nvskills-signing and others added 2 commits August 17, 2026 13:26
Signed-off-by: nvskills-svc-account <svc-nvskills-signing@nvidia.com>
Replace cache discovery and normalization helpers with one declarative stage-and-verify pass.

Co-authored-by: Codex <noreply@openai.com>
Signed-off-by: Wenqi Li <wenqil@nvidia.com>
@wyli

wyli commented Aug 17, 2026

Copy link
Copy Markdown
Collaborator Author

/nvskills-ci

2 similar comments
@wyli

wyli commented Aug 17, 2026

Copy link
Copy Markdown
Collaborator Author

/nvskills-ci

@wyli

wyli commented Aug 19, 2026

Copy link
Copy Markdown
Collaborator Author

/nvskills-ci

Signed-off-by: nvskills-svc-account <svc-nvskills-signing@nvidia.com>
@wyli
wyli marked this pull request as ready for review August 19, 2026 10:49
@wyli wyli changed the title Add MLflow finetuning demo Add MLflow tracking and harden NV-Segment-CT fine-tuning Aug 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants