clusterd-test-driver: add explain verb to assert optimized plan shape

[antiguru]

Follow-up to #37008 (now merged), addressing DAlperin's review on join.spec: a script using optimize asserts only the result, so optimizer or lowering drift could silently change the plan under test.

explain renders the lowered LIR plan (the EXPLAIN PHYSICAL PLAN form, via a no-catalog DummyHumanizer so ids are u<n> and columns #n) as its golden, submitting nothing. It takes the dataflow either inline (the create-dataflow body) or by reference — explain ref=<name> renders a dataflow a prior create-dataflow name=<name> declared, without repeating its body. join.spec declares the join, then explain ref=join asserts the differential-join plan alongside the count.

The multi-object render separates objects with blank lines, so the .spec format gains the datadriven doubled----- block form, emitted automatically by REWRITE.

🤖 Generated with Claude Code

[def-]

QA LLM Review

1. MEDIUM -- explain inherits ExplainConfig::default(), whose redacted flag follows the build profile

src/clusterd-test-driver/src/dataflow.rs:467

ExplainConfig::default() sets redacted: !mz_ore::assert::soft_assertions_enabled(), so an explain golden is rendered with literals redacted to █ in exactly the builds the scripts run in, and un-redacted under cargo test / PROFILE=dev / MZ_SOFT_ASSERTIONS=1. Any explain golden that contains a literal will both hide the constant it means to assert and differ between two supported ways of producing it.

Details

The chain: src/repr/src/explain.rs:212 derives redacted from soft_assertions_enabled(), and SOFT_ASSERTIONS (src/ore/src/assert.rs:66) defaults to cfg!(debug_assertions) unless MZ_SOFT_ASSERTIONS is set. The driver image is a plain cargo-build pre-image (src/clusterd-test-driver/ci/mzbuild.yml) at mzbuild's optimized/release profile, and HeadlessDriver (test/clusterd-test-driver/mzcompose.py:47) sets no MZ_SOFT_ASSERTIONS, so both the mzcompose run and run-local.py's default PROFILE=optimized render with redacted = true. The new explain_join_renders_plan unit test, running under cargo test, renders with redacted = false.

Nothing differs today: join.spec's plan has no literals, only ids, Arrange/Stream nodes and column keys. The first explain over a plan with a Filter, a Constant, a TopK limit or a non-column arrangement key gets Filter (#0 = █) instead of the value, silently unassertable — and a golden REWRITEn under PROFILE=dev will not match the mzcompose run. The doc comment's claim that the render is "stable" holds for ids and column references only.

Fix:

// Golden output must not depend on the build's soft-assertion setting, which
// `ExplainConfig::default()` derives `redacted` from.
let config = ExplainConfig {
    redacted: false,
    ..ExplainConfig::default()
};

[antiguru]

Confirmed, and fixed in #38483 (this PR had already merged).

Reproduced the divergence: with MZ_SOFT_ASSERTIONS=0, a plan with a literal renders map=(█); with soft assertions on it renders the value. The fix pins redacted: false in an explain_config helper, so the render depends only on the plan and not on the build profile, plus a test that goes red without the pin.

_{Posted by Claude Code on behalf of @antiguru}