Skip to content

Make explicitly selected Qwen model gates fail closed #942

Description

@xiaguan

Problem

Several Qwen model gates combine #[ignore] with optional fixture helpers. Explicitly selecting the gate can therefore return success without running the model path.

At 69c4e149:

  • Qwen3 has five ignored model gates. The two LoRA smokes fail when their model fixture is unavailable, but the three DFlash gates call target_path_or_skip() / draft_path_or_skip() and return early.
  • Qwen3.5 has eleven ignored gates. They share tests/common/model_fixture.rs; a missing or empty environment value, unreadable or invalid config.json, or a non-Qwen3.5 config returns None.
  • Qwen3.5 HF gates can also return early when the local model revision is unknown.

For example, with the model variables unset, these commands can report success after only printing SKIP messages:

cargo test --release -p pegainfer-qwen3 \
  --test dflash_sampled_equivalence -- --ignored --nocapture

cargo test --release -p pegainfer-qwen35 --features qwen35 \
  --lib -- --ignored --nocapture

Both crates are already compiled and Clippy-checked with all targets in CI, which protects the type surface. It does not prove that a selected checkpoint/GPU gate executed.

Direction

Keep optional fixture lookup for ordinary test runs on machines without model weights. Add a required fixture path for explicitly selected model gates. Once a maintainer asks for one of these gates, missing or invalid prerequisites are failures rather than skips.

Provide one named command per model line that lists the exact maintained gates instead of relying on an unscoped repository-wide --ignored run.

Acceptance criteria

  • Qwen3's three ignored DFlash gates fail when either required fixture is missing or invalid.
  • Qwen3.5's eleven ignored gates fail when the model fixture is missing, empty, unreadable, invalid JSON, the wrong model type, or lacks the required pinned revision for an HF gate.
  • The normal CPU-only test path can still skip model-backed tests when they were not explicitly selected.
  • A checked-in Qwen3 runner names the DFlash and LoRA gates it executes.
  • A checked-in Qwen3.5 runner names all eleven maintained ignored gates across library and integration-test targets.
  • Each runner prints the source commit plus selected/completed gate counts and exits non-zero on any skip.
  • The existing feature-on all-target check and Clippy gates remain green.

A GPU GitHub runner is not required by this issue. The goal is to make maintainer-run evidence unambiguous.

Contributor guardrails

  • The PR description must show the current false-green reproduction and the same missing-prerequisite case failing after the change.
  • Include one successful real-model runner transcript for every model line changed, with exact commit and selected/completed counts. A mock model, fake CUDA layer, or helper-only unit test is not a substitute.
  • Make the selected-gate path fail closed in code. Do not claim completion by grepping logs for SKIP while the test process still reports success.
  • Keep the gate manifest explicit. Do not discover tests by source grep, naming convention, or an unbounded repository-wide --ignored invocation.
  • Do not add a new generic test framework, public API, or compatibility wrapper for this issue.
  • Map every changed file to an acceptance criterion in the PR description; unrelated refactors and test rewrites are out of scope.

Metadata

Metadata

Assignees

Labels

bugSomething isn't workingqwen3Qwen3-4B model crate (pegainfer-qwen3-4b)qwen35Qwen3.5-4B model crate (pegainfer-qwen35-4b)

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions