feat: surface partial-run status from dropped engines (PERFNFV-464)

[k-rister]

Summary

• Completes the crucible-side piece of PERFNFV-464 (dropped-engine detection) — CDM (PR#204), toolbox (PR#129), and rickshaw (PR#863) are already merged.
• log_result_directory() in bin/result-processor.py now prints partial: yes (N engine(s) dropped) alongside the existing status: line when a run's rickshaw-run.json has partial: true.
• Adds a "Partial runs and dropped engines" section to docs/how-benchmark-execution-works.md covering when engines get dropped, the different outcomes for per-sample vs. post-benchmark drops, and how it surfaces in rickshaw-run.json, the CLI, and CDM.
• Updates docs/how-cdm-works.md's run doctype field list to include partial/dropped-engines.

Test plan

• crucible ls --result-dir <clean run> — no partial: line
• crucible ls --result-dir <run with a dropped engine> — shows partial: yes (1 engine(s) dropped)
• python3 -m py_compile bin/result-processor.py

🤖 Generated with Claude Code

[k-rister]

PR Review: crucible#652 — feat: surface partial-run status from dropped engines (PERFNFV-464)

Summary: Adds a partial: yes (N engine(s) dropped) line to crucible ls/crucible get result output when a run's rickshaw-run.json has partial: true, and documents the dropped-engine/partial-run feature (completing the crucible-side piece of PERFNFV-464, after CDM/toolbox/rickshaw already merged their parts).

Changed files: 3
Review dimensions: Correctness, API & Contracts, Build & Deploy, Documentation, Style, Completeness

Verified the code against the merged rickshaw implementation (state.run["partial"]/state.run["dropped-engines"] written as top-level keys in rickshaw-run.json, matching subprojects/core/rickshaw/schema/rickshaw-run.json) and against the merged CDM schema change (v9dev+ run.partial/run.dropped-engines). Both call sites of log_result_directory() are updated consistently, and the doc's description of per-sample vs. post-benchmark drop behavior matches rickshaw-run.py's evaluate_test_roadblock()/process_bench_roadblocks() logic exactly.

Documentation

• [docs/how-benchmark-execution-works.md:505-507] Claims crucible get result surfaces the partial: yes (N engine(s) dropped) line — it doesn't. crucible get result (bin/base:821 get_result() → get_result_backend()) invokes get-result-summary.sh/.js in CommonDataModel, which queries the CDM server and has no reference to partial/dropped-engines/status. Only crucible ls (which invokes bin/result-processor.py, the file actually changed here) goes through the new log_result_directory() code path. A user following this doc and running crucible get result to check for partial-run status won't see it.
• [docs/how-cdm-works.md:416] Versioning table wasn't updated for the new v9dev fields, though the Document types table in the same file was. The PR adds partial, dropped-engines to the "Document types" table (line 44) but doesn't add a corresponding note to the "Versioning" table's v9dev row, unlike the precedent set by commit bac30d7 (which updated both tables together when default-aggregation was added to v10dev).

File Coverage

• bin/result-processor.py — No issues found
• docs/how-benchmark-execution-works.md — 1 doc issue
• docs/how-cdm-works.md — 1 doc issue

Limitations

Did not execute the actual CLI against a live run directory (relied on the PR author's stated test plan and static code tracing). Did not verify runtime behavior of the rickshaw/CDM/toolbox pieces since those are already-merged, out-of-scope repos.

Verdict

Approve with comments — the code change is correct and consistent with the already-merged rickshaw/CDM work; both findings are doc-only inaccuracies/gaps that don't block merging but should be fixed (the get result claim is a factual error a user could act on).

🤖 Generated with Claude Code

[k-rister]

Thanks for the review — took the two findings in different directions per feedback:

#1 (crucible get result doesn't actually surface partial status): rather than walking back the doc's claim, made it true. crucible get result → bin/base's get_result() → get_result_backend() → CommonDataModel's get-result-summary.sh/.js, a completely separate code path from bin/result-processor.py. Added a new GET /api/v1/run/:id/partial-status endpoint to the CDM query server (mirrors the existing /benchmark endpoint, reads the run.partial/run.dropped-engines fields already in the v9dev+ mapping via the existing cdm.getRunData() helper), and wired get-result-summary.js to fetch it and print partial: yes (N engine(s) dropped) — same wording bin/result-processor.py already uses for crucible ls. Verified end-to-end: the partial: false path against a real indexed run, and partial: true against a synthetic run+iteration doc pair inserted directly into OpenSearch and deleted afterward (getting a real dropped-engine run to survive indexing is its own very specific failure mode, so no naturally-occurring partial: true doc was available to test against). Opened as CommonDataModel PR#206: perftool-incubator/CommonDataModel#206 — the doc's claim here becomes accurate once that merges.

#2 (Versioning table gap): fixed directly, 77c47d5 — added partial/dropped-engines to the v9dev row, matching the bac30d7 precedent of keeping both tables in sync.

🤖 Generated with Claude Code

[atheurer]

The Python logic looks correct and well-structured. One issue worth fixing before merge:

Documentation inaccuracy in docs/how-benchmark-execution-works.md

The new section states that both crucible ls and crucible get result surface partial status as a partial: yes (N engine(s) dropped) line. However, crucible get result routes through get-result-summary.sh (a CDM query script), not result-processor.py, and this PR makes no changes to that path. The formatted partial: yes line only appears in crucible ls output.

Either drop the crucible get result reference from that sentence, or clarify that CDM queries return the raw partial/dropped-engines fields in their own format (via CDM PR#206).

Minor items (not blockers):

• If partial: true ever appears without a dropped-engines key, the output reads partial: yes (0 engine(s) dropped) which is self-contradictory. Adding and data.get('dropped-engines') to the guard at line 308 would prevent this.
• Test plan covers crucible ls --result-dir but not crucible tags ls --result-dir, though run_results_tag_mode() (line 698) was also updated.

[k-rister]

This issue is actually being addressed in perftool-incubator/CommonDataModel#206