Skip to content

flaky: prefill_q4_seq4_synthetic_smoke fails only under concurrent build/GPU load #198

Description

@chrishayuk

prefill_q4_seq4_synthetic_smoke fails only under concurrent build/GPU load.

Test

crates/larql-compute-metal/tests/test_metal_decode_synthetic.rs:762

assert_eq!(result.len(), seq_len * HIDDEN, "prefill_kquant output length");

The failure is assertion left == right failed on the output-length check
immediately after prefill_kquant returns Some, so the kernel produced a
buffer of the wrong size rather than declining (None is handled above and
skips the test).

Reproduction rates (Apple Silicon, local, 2026-08-02)

condition failures
cargo test -p larql-compute-metal, concurrent cargo build of another crate 2 / ~5
cargo test -p larql-compute-metal, otherwise-idle machine 0 / 10
cargo test -p larql-compute-metal --test test_metal_decode_synthetic (isolated binary) 0 / 13
cargo test -p larql-compute-metal -- --test-threads=1 0 / 5
same package on main-equivalent HEAD, package level 0 / 2

Concurrency context

cargo test -p <pkg> runs each test binary as a separate process, and the
package has ~15 of them; several drive Metal. The failing case is therefore
multiple processes contending for the GPU plus rustc/linker load. Isolating
the binary or serialising test threads within it did not reproduce, and
neither did the package-level run once the machine was idle — so the trigger
looks like total concurrent load rather than intra-binary thread interleaving.

Not yet established: whether serial execution eliminates it or merely lowers
the rate. 5 clean serial runs on an idle machine is not evidence, because the
package-level runs were also clean under the same conditions. The discriminating
experiment is serial + concurrent build load, which has not been run.

Not caused by the strict-refusal work

Noticed while running cargo test --workspace during PR #197's follow-on.
larql-compute-metal has no dependency on larql-kv, cargo test -p larql-compute-metal builds and runs only that package's binaries, and the diff
was confined to crates/larql-kv/**. Recording it here rather than letting it
become known-flake folklore.

Related prior art: the threadgroup-scratch reuse race in fused attention
(fixed), which had a similar load-dependent signature.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions