feat(cli): parallel catalog validation and fleet rollup - #123
feat(cli): parallel catalog validation and fleet rollup#123mimran-khan wants to merge 21 commits into
Conversation
Emit a machine-readable fleet rollup at the reports root with per-skill status, optional severity totals from child JSON reports, and report paths. Create the output directory when needed so summary writes survive early skill failures. Fixes NVIDIA#120 Signed-off-by: mimran-khan <mohammed_imran.khan@outlook.com>
Catalog validate accepts --workers N to run skills in isolated child processes. Values above 1 skip the per-skill pipeline view and rebuild per-skill argv from the parent Click context or sys.argv. Fixes NVIDIA#122 Signed-off-by: mimran-khan <mohammed_imran.khan@outlook.com>
Rebuild child argv without dropping positional catalog paths, track fresh per-skill JSON reports instead of stale files, write catalog-summary.json atomically, and fix --include-skills forwarding for context fallback. Signed-off-by: mimran-khan <mohammed_imran.khan@outlook.com>
|
Worker argv, stale JSON, atomic summary, and --include-skills fixes are pushed. Ready for re-review. |
|
@mimran-khan : Re-reviewed at current head
The atomic summary writer and |
rng1995
left a comment
There was a problem hiding this comment.
Parallel catalog execution still loses real CLI option values, changes explicit CLI-only reporting, and can reuse stale per-skill JSON after a worker produces no report. Focused catalog tests and Ruff/diff checks passed; the single Python 3.12 CI failure is an unrelated flaky NVIDIA bridge socket test. Requesting changes for the reproducible worker-boundary defects in the review threads.
Rebuild child argv from Click params, preserve -r cli when selected, track per-run JSON in serial catalog mode, and stop attaching stale reports when no new JSON was produced this run. Signed-off-by: mimran-khan <mohammed_imran.khan@outlook.com>
# Conflicts: # CHANGELOG.md
Signed-off-by: mimran-khan <mohammed_imran.khan@outlook.com>
Do not forward the implicit cli default to catalog workers, and ignore SARIF sidecars when picking the standard JSON report for fleet rollup. Signed-off-by: mimran-khan <mohammed_imran.khan@outlook.com>
Signed-off-by: mimran-khan <mohammed_imran.khan@outlook.com>
Record the JSON artifact emitted by each validate invocation instead of rediscovering reports via directory diffs, which missed same-second overwrites in parallel catalog mode. Also stop pre-creating catalog summary parents through symlinked paths.
|
|
||
|
|
||
| def _record_validate_json_report(report_name: str | None) -> None: | ||
| global _last_validate_json_report |
There was a problem hiding this comment.
[P1] Remove the module-global report handoff. The required Python 3.12 job deterministically fails here and at line 348 because Ruff reports PLW0603; process-global state also makes nested or re-entrant validate calls share provenance. Please return the emitted JSON basename through the Click invocation and worker result, or store it in invocation-scoped context state, instead of mutating _last_validate_json_report.
There was a problem hiding this comment.
The ContextVar change fixes the process-global handoff, and the focused suite plus a real parallel invocation pass. The requested cross-invocation isolation regression is still missing, however, and DCO still fails for unsigned commits fb87d4a and 21cdb16. Please add the regression and repair the sign-offs before this thread can be resolved.
rng1995
left a comment
There was a problem hiding this comment.
The previously reported worker argument, report-format, stale-report, SARIF-selection, and same-second overwrite defects are fixed at the current head, and all 53 focused command tests pass. Required CI is still red: Ruff rejects the new module-global report handoff, and DCO reports fb87d4a and 21cdb16 as unsigned. Please replace the global with invocation-scoped state, fix the sign-offs, and rerun the checks.
Replace module-global report name slot so parallel catalog workers and Ruff PLW0603 stay clean while preserving exact JSON binding after validate. Signed-off-by: mimran-khan <mohammed_imran.khan@outlook.com>
Signed-off-by: mimran-khan <mohammed_imran.khan@outlook.com>
|
@mimran-khan, this PR is still blocked at |
Signed-off-by: mimran-khan <mohammed_imran.khan@outlook.com>
Fixes #122
Related: #120, #121
This branch includes two catalog features that stack cleanly:
catalog-summary.json([FEA]: Machine-readable catalog fleet rollup report #120 / feat(cli): write catalog-summary.json after catalog validate #121): fleet rollup after catalog validate--workers N([FEA]: Parallel catalog validation with --workers #122): parallel child-process validation for catalogs--workersdefaults to 1 (serial, same per-skill pipeline view as today). Above 1, skills validate in isolated processes with per-skill output dirs under-o. The parent Click context (orsys.argvwhen launched from the real CLI) is used to rebuild each childvalidateinvocation, so pytest and production entry points both work.Parallel mode skips the per-skill Rich pipeline view and prints a single fleet scoreboard at the end. Exit code stays nonzero if any skill failed.
If #121 lands first, I can rebase this to workers-only; happy to split either way.
Test plan
pytest tests/test_commands.py -k catalog--workers 1(default)--workers 2integration test on two fixture skills