refactor(qwen3): move the report harness out of the library API

docs/models/qwen3/roadmap.md

@@ -2,7 +2,7 @@
 
 > **TL;DR:** Qwen3-4B is the maturity bar of the project — continuous batching, TP=2, default-on prefix cache (#216), and the HF logits golden gate are all live — so its roadmap is sharpening, not bring-up. The #220 RoPE OOB bug is fixed (cos/sin cache sized from `max_position_embeddings`, admission rejects past the window, kernel traps an out-of-range position; gated by both an oversized-reject and an in-window >4096 IT). Since the 2026-06-04 open-set review, batched greedy sampling (#307), mixed greedy/non-greedy batched sampling (#284), and in-process pegaflow KV offload / L2 host tier (#316, pure-L2 TTFT 195→40ms) have landed in the table below. Remaining open set: zero TP correctness coverage, LoRA gated only by a zero-adapter smoke, prefix-cache observability dropped at the scheduler boundary, stale docs, and the YaRN #8 follow-up for rope-scaled checkpoints.
 >
-> **Last touched:** 2026-06
+> **Last touched:** 2026-08
 
 Tracking issue: see the `[Model] Qwen3-4B roadmap` GitHub issue. Cross-model items stay in `docs/roadmap/execution.md`; this doc owns the qwen3 line.
 
@@ -48,7 +48,7 @@ Tracking issue: see the `[Model] Qwen3-4B roadmap` GitHub issue. Cross-model ite
 
 - **Issue hygiene:** #188 references a test target deleted in #194 — close as superseded by the golden gate. #203 §1 still claims qwen3 has no prefix reuse — stale since #216.
 - **Dead code:** ~~`batch_decode_trace.rs` `HIDDEN_SIZE`/`INTERMEDIATE_SIZE` consts; qwen3 `probe_model()`+`ModelInfo`~~ — removed in the hawk dead-public sweep (#743).
-- **File size:** `executor.rs` (1435), `scheduler.rs` (1420, ~826 of them inline tests), `kernel_bench.rs` (1112) breach the 1k-line redline.
+- **File size:** `executor.rs` (1435) and `scheduler.rs` (1420, ~826 of them inline tests) breach the 1k-line redline. ~~`kernel_bench.rs` (1112)~~ — the report harness left the library in #944; it now lives under `src/bin/report_support/`, split across five modules and compiled only by the `kernel-report` binaries.
 - **Docs:** `model-crate.md` TL;DR advertises a deleted `qwen3_kernel_snapshot` bench and, with `kernels-crate.md`, uses the obsolete `crates/` layout in every command — collapse both into one slim layout doc. `tp-design.md` describes the implemented controller/worker runtime as future direction — rewrite to past tense, promote the 3 real open items. `kv-pressure-hang.md` — lift the KV-lifetime-reservation lessons to `docs/lessons/`, then delete. `execution.md` Done list predates #216.
 
 ## Done criteria

docs/subsystems/kernels/kernel-op-reports.md

@@ -1,9 +1,9 @@
 # Kernel Op Reports
 
 **Created**: 2026-05-04
-**Last touched**: 2026-06
+**Last touched**: 2026-08
 **Status**: active; prefill/report commit ready, decode tuning deferred
-**TL;DR**: `qwen3_kernel_snapshot` is no longer a Cargo bench. Qwen3 has feature-gated `qwen3_kernel_report` per-op kernel tooling and `qwen3_model_report` model-level decode operator reporting. Decode now routes through an eager `BatchDecodeDag`, so the executable forward sequence is also the `KernelCall` contract source for runtime tracing; `qwen3_model_report` disables CUDA Graph, traces that DAG, then joins traced TensorSpecs against measured microbench results to emit by-op, by-call-site, coverage, schedule-preview, latency-stat, and Graphviz DOT reports. Prefill remains covered by `qwen3_kernel_report` stage reports; measured FA2 `CTA_TILE_Q=64` is the Qwen3 production prefill default on the RTX 5090 grid. The model-agnostic timing loop, latency stats, and by-op/by-call-site rollup now live in the `pegainfer-bench` crate, shared by the Kimi-K2 model report; the qwen3 attention-specific regression framework (manifest/snapshot/provenance/CUPTI/cold-L2) deliberately stays in `qwen3_kernel_report`. `qwen3_model_report` traces + measures projection GEMMs under the production `numeric_policy()`: Pin and PerToken are faithful (Pin's serving counter-observed via `pin_served`, PerToken's via `per_token_served`), while under the default Tuned policy the GEMM rows are classified `unfaithful_gemmex` and excluded from the totals without being measured (the untuned context would run GemmEx, not production's per-process-tuned algo); the JSON config records the numeric policy and a partial-total flag.
+**TL;DR**: The harness source moved out of the library in #944 — it now lives in `pegainfer-qwen3/src/bin/report_support/` and is compiled only by the two `kernel-report` binaries, so paths in the Execution Log below are historical. `qwen3_kernel_snapshot` is no longer a Cargo bench. Qwen3 has feature-gated `qwen3_kernel_report` per-op kernel tooling and `qwen3_model_report` model-level decode operator reporting. Decode now routes through an eager `BatchDecodeDag`, so the executable forward sequence is also the `KernelCall` contract source for runtime tracing; `qwen3_model_report` disables CUDA Graph, traces that DAG, then joins traced TensorSpecs against measured microbench results to emit by-op, by-call-site, coverage, schedule-preview, latency-stat, and Graphviz DOT reports. Prefill remains covered by `qwen3_kernel_report` stage reports; measured FA2 `CTA_TILE_Q=64` is the Qwen3 production prefill default on the RTX 5090 grid. The model-agnostic timing loop, latency stats, and by-op/by-call-site rollup now live in the `pegainfer-bench` crate, shared by the Kimi-K2 model report; the qwen3 attention-specific regression framework (manifest/snapshot/provenance/CUPTI/cold-L2) deliberately stays in `qwen3_kernel_report`. `qwen3_model_report` traces + measures projection GEMMs under the production `numeric_policy()`: Pin and PerToken are faithful (Pin's serving counter-observed via `pin_served`, PerToken's via `per_token_served`), while under the default Tuned policy the GEMM rows are classified `unfaithful_gemmex` and excluded from the totals without being measured (the untuned context would run GemmEx, not production's per-process-tuned algo); the JSON config records the numeric policy and a partial-total flag.
 
 ## Preparation
 

pegainfer-qwen3/src/batch_decode_buffers.rs

@@ -38,8 +38,8 @@ const DECODE_ATTENTION_PATH_COUNT: usize = 2;
 // already saturate the SMs (bs<=8 wins big, ~bs16 even, bs32 within ~1%).
 // 64-token chunks measured fastest on RTX 5090 (128/256 are 1-7% slower, 32
 // past the merge-overhead knee). Measurements: docs/models/qwen3/decode-attention.md.
-pub(crate) const SPLIT_KV_CHUNK_TOKENS: usize = 64;
-pub(crate) const SPLIT_KV_TUNED_MAX_CHUNKS: usize = 64; // Tuned adaptive-split count cap
+pub const SPLIT_KV_CHUNK_TOKENS: usize = 64;
+pub const SPLIT_KV_TUNED_MAX_CHUNKS: usize = 64; // Tuned adaptive-split count cap
 const SPLIT_KV_MAX_CHUNKS_PER_REQUEST: usize = 256; // split-KV workspace/guard bound
 const SPLIT_KV_MAX_BATCH_SIZE: usize = 32;
 

pegainfer-qwen3/src/bin/qwen3_kernel_report.rs

@@ -19,39 +19,43 @@ use clap::Subcommand;
 use cudarc::driver::sys;
 use pegainfer_cupti::profile_range_with_prepare;
 use pegainfer_kernels::tensor::DeviceContext;
-use pegainfer_qwen3::kernel_bench::AttentionDecodeCase;
-use pegainfer_qwen3::kernel_bench::AttentionKernelShape;
-use pegainfer_qwen3::kernel_bench::AttentionKernelSpec;
-use pegainfer_qwen3::kernel_bench::AttentionKernelVariant;
-use pegainfer_qwen3::kernel_bench::AttentionPrefillCase;
-use pegainfer_qwen3::kernel_bench::DecodePath;
-use pegainfer_qwen3::kernel_bench::DenseCase;
-use pegainfer_qwen3::kernel_bench::DenseKernelKind;
-use pegainfer_qwen3::kernel_bench::DevicePeakBandwidth;
-use pegainfer_qwen3::kernel_bench::GemmProjection;
-use pegainfer_qwen3::kernel_bench::HEAD_DIM;
-use pegainfer_qwen3::kernel_bench::HIDDEN_SIZE;
-use pegainfer_qwen3::kernel_bench::INTERMEDIATE_SIZE;
-use pegainfer_qwen3::kernel_bench::L2CacheClear;
-use pegainfer_qwen3::kernel_bench::NUM_KV_HEADS;
-use pegainfer_qwen3::kernel_bench::NUM_QO_HEADS;
-use pegainfer_qwen3::kernel_bench::PAGE_SIZE;
-use pegainfer_qwen3::kernel_bench::PrefillAttentionShape;
-use pegainfer_qwen3::kernel_bench::PrefillAttentionSpec;
-use pegainfer_qwen3::kernel_bench::PrefillAttentionVariant;
-use pegainfer_qwen3::kernel_bench::PrefillStage;
-use pegainfer_qwen3::kernel_bench::REPORT_ITERS;
-use pegainfer_qwen3::kernel_bench::SinglePrefillCase;
-use pegainfer_qwen3::kernel_bench::SplitKvConfig;
-use pegainfer_qwen3::kernel_bench::VOCAB_SIZE;
-use pegainfer_qwen3::kernel_bench::cache_clear_bytes;
+use pegainfer_qwen3::runtime::SplitKvConfig;
 use serde::Deserialize;
 use serde::Serialize;
 use serde_json::Value;
 use serde_json::json;
 use sha2::Digest;
 use sha2::Sha256;
 
+#[path = "report_support/mod.rs"]
+mod report_support;
+
+use report_support::common::DevicePeakBandwidth;
+use report_support::common::HEAD_DIM;
+use report_support::common::L2CacheClear;
+use report_support::common::NUM_KV_HEADS;
+use report_support::common::NUM_QO_HEADS;
+use report_support::common::PAGE_SIZE;
+use report_support::common::REPORT_ITERS;
+use report_support::common::cache_clear_bytes;
+use report_support::decode_attention::AttentionDecodeCase;
+use report_support::decode_attention::AttentionKernelShape;
+use report_support::decode_attention::AttentionKernelSpec;
+use report_support::decode_attention::AttentionKernelVariant;
+use report_support::decode_attention::DecodePath;
+use report_support::dense::DenseCase;
+use report_support::dense::DenseKernelKind;
+use report_support::dense::GemmProjection;
+use report_support::dense::HIDDEN_SIZE;
+use report_support::dense::INTERMEDIATE_SIZE;
+use report_support::dense::VOCAB_SIZE;
+use report_support::prefill_attention::AttentionPrefillCase;
+use report_support::prefill_attention::PrefillAttentionShape;
+use report_support::prefill_attention::PrefillAttentionSpec;
+use report_support::prefill_attention::PrefillAttentionVariant;
+use report_support::prefill_attention::PrefillStage;
+use report_support::single_prefill::SinglePrefillCase;
+
 const DEFAULT_MANIFEST: &str = concat!(env!("CARGO_MANIFEST_DIR"), "/kernel_manifests/qwen3.toml");
 const DEFAULT_OP: &str = "paged_decode_attention";
 const PREFILL_OP: &str = "paged_prefill_attention";

pegainfer-qwen3/src/bin/qwen3_model_report.rs

@@ -37,6 +37,7 @@ use pegainfer_kernels::ops::per_token_served;
 use pegainfer_kernels::ops::pin_served;
 use pegainfer_kernels::ops::reset_numeric_policy_counters;
 use pegainfer_kernels::ops::set_numeric_policy;
+use pegainfer_kernels::ops::split_qkv_into;
 use pegainfer_kernels::tensor::DeviceContext;
 use pegainfer_kernels::tensor::DeviceVec;
 use pegainfer_kernels::tensor::HiddenStates;
@@ -52,10 +53,17 @@ use pegainfer_qwen3::batch_decode_trace::PHASE_DECODE;
 use pegainfer_qwen3::batch_decode_trace::RMS_NORM_EPS;
 use pegainfer_qwen3::batch_decode_trace::normalize_call_site;
 use pegainfer_qwen3::batch_decode_trace::trace_decode_kernel_calls;
-use pegainfer_qwen3::kernel_bench::L2CacheClear;
-use pegainfer_qwen3::kernel_bench::build_split_kv_csr;
 use serde::Serialize;
 
+// Only `L2CacheClear` is needed here; the rest of the `report_support` tree
+// belongs to `qwen3_kernel_report`. The allow covers the sibling helpers in
+// `common` that this binary does not call.
+#[allow(dead_code)]
+#[path = "report_support/common.rs"]
+mod common;
+
+use common::L2CacheClear;
+
 const DEFAULT_ITERS: u64 = 32;
 
 #[derive(Parser)]
@@ -477,14 +485,13 @@ fn measure_catalog(
         if catalog.contains_key(&key) {
             continue;
         }
-        let measure = match classify(policy, &call.op) {
-            Some(reason) => Measure::Excluded(reason),
-            None => {
-                let stats = measure_call(call, iters).with_context(|| {
-                    format!("failed to measure {}\n{}", call.label, describe_call(call))
-                })?;
-                Measure::Faithful(stats)
-            }
+        let measure = if let Some(reason) = classify(policy, &call.op) {
+            Measure::Excluded(reason)
+        } else {
+            let stats = measure_call(call, iters).with_context(|| {
+                format!("failed to measure {}\n{}", call.label, describe_call(call))
+            })?;
+            Measure::Faithful(stats)
         };
         catalog.insert(key.clone(), BenchEntry { key, measure });
     }
@@ -632,7 +639,7 @@ fn measure_split_qkv(call: &KernelCall, iters: u64) -> Result<LatencyStats> {
     let mut k = HiddenStates::zeros(&ctx, kv_dim, batch)?;
     let mut v = HiddenStates::zeros(&ctx, kv_dim, batch)?;
     measure_loop(&ctx, iters, || {
-        ops::split_qkv_into(&ctx, &qkv, &mut q, &mut k, &mut v)?;
+        split_qkv_into(&ctx, &qkv, &mut q, &mut k, &mut v)?;
         Ok(())
     })
 }
@@ -715,7 +722,8 @@ fn measure_paged_decode_attention(call: &KernelCall, iters: u64) -> Result<Laten
         let split_chunk_size = attr_usize(call, "split_chunk_size")?;
         let cap = attr_usize(call, "split_max_chunks")?;
         let padded_slots = batch * cap;
-        let split_csr = build_split_kv_csr(split_chunk_size, cap, &vec![kv_len; batch], batch)?;
+        let split_csr =
+            ops::build_split_kv_csr(split_chunk_size, cap, &vec![kv_len; batch], batch)?;
         let split_kv_chunk_size = [split_chunk_size as i32];
         let split_request_indices_d = ctx.stream.clone_htod(&split_csr.request_indices)?;
         let split_kv_tile_indices_d = ctx.stream.clone_htod(&split_csr.kv_tile_indices)?;

pegainfer-qwen3/src/bin/report_support/common.rs

@@ -0,0 +1,114 @@
+//! Report-harness primitives shared across every benchmark domain: device
+//! geometry constants, L2 cache clearing, device bandwidth query, and
+//! synthetic buffer generation.
+
+use std::mem::size_of;
+
+use anyhow::Result;
+use anyhow::anyhow;
+use cudarc::driver::CudaSlice;
+use cudarc::driver::DevicePtr;
+use cudarc::driver::DevicePtrMut;
+use cudarc::driver::sys;
+use half::bf16;
+use pegainfer_kernels::ffi;
+use pegainfer_kernels::tensor::DeviceContext;
+use serde::Serialize;
+
+pub(crate) const NUM_LAYERS: usize = 1;
+pub(crate) const NUM_QO_HEADS: usize = 32;
+pub(crate) const NUM_KV_HEADS: usize = 8;
+pub(crate) const HEAD_DIM: usize = 128;
+pub(crate) const PAGE_SIZE: usize = 16;
+pub(crate) const REPORT_ITERS: u64 = 128;
+const MEMORY_TRANSFERS_PER_CLOCK: f64 = 2.0;
+const CACHE_CLEAR_L2_MULTIPLIER: usize = 2;
+const CACHE_CLEAR_MIN_BYTES: usize = 128 * 1024 * 1024;
+
+#[derive(Clone, Copy, Debug, Serialize)]
+pub(crate) struct DevicePeakBandwidth {
+    pub(crate) memory_clock_khz: i32,
+    pub(crate) memory_bus_width_bits: i32,
+    peak_bytes_per_sec: f64,
+}
+
+impl DevicePeakBandwidth {
+    pub(crate) fn query(ctx: &DeviceContext) -> Result<Self> {
+        let memory_clock_khz = ctx
+            .ctx
+            .attribute(sys::CUdevice_attribute::CU_DEVICE_ATTRIBUTE_MEMORY_CLOCK_RATE)
+            .map_err(|e| anyhow!("failed to query memory clock: {e}"))?;
+        let memory_bus_width_bits = ctx
+            .ctx
+            .attribute(sys::CUdevice_attribute::CU_DEVICE_ATTRIBUTE_GLOBAL_MEMORY_BUS_WIDTH)
+            .map_err(|e| anyhow!("failed to query memory bus width: {e}"))?;
+        let peak_bytes_per_sec = f64::from(memory_clock_khz)
+            * 1_000.0
+            * (f64::from(memory_bus_width_bits) / 8.0)
+            * MEMORY_TRANSFERS_PER_CLOCK;
+
+        Ok(Self {
+            memory_clock_khz,
+            memory_bus_width_bits,
+            peak_bytes_per_sec,
+        })
+    }
+
+    pub(crate) fn peak_gb_per_sec(&self) -> f64 {
+        self.peak_bytes_per_sec / 1.0e9
+    }
+}
+
+pub(crate) struct L2CacheClear {
+    a: CudaSlice<bf16>,
+    b: CudaSlice<bf16>,
+    out: CudaSlice<bf16>,
+    len: usize,
+}
+
+impl L2CacheClear {
+    pub(crate) fn new(ctx: &DeviceContext) -> Result<Self> {
+        let l2_bytes =
+            ctx.ctx
+                .attribute(sys::CUdevice_attribute::CU_DEVICE_ATTRIBUTE_L2_CACHE_SIZE)
+                .map_err(|e| anyhow!("failed to query L2 cache size: {e}"))? as usize;
+        let clear_bytes = cache_clear_bytes(l2_bytes);
+        let len = clear_bytes.div_ceil(size_of::<bf16>());
+
+        Ok(Self {
+            a: ctx.stream.alloc_zeros(len)?,
+            b: ctx.stream.alloc_zeros(len)?,
+            out: ctx.stream.alloc_zeros(len)?,
+            len,
+        })
+    }
+
+    pub(crate) fn clear(&mut self, ctx: &DeviceContext) -> Result<()> {
+        // CUDA's reset-persisting-L2 APIs do not evict normal cache lines, so
+        // benchmarks use a large streaming kernel to push prior data out of L2.
+        let (a_ptr, _a_guard) = self.a.device_ptr(&ctx.stream);
+        let (b_ptr, _b_guard) = self.b.device_ptr(&ctx.stream);
+        let (out_ptr, _out_guard) = self.out.device_ptr_mut(&ctx.stream);
+        let result = unsafe {
+            ffi::add_cuda(
+                a_ptr as *const ffi::Half,
+                b_ptr as *const ffi::Half,
+                out_ptr as *mut ffi::Half,
+                self.len as i32,
+                ctx.stream.cu_stream(),
+            )
+        };
+        result.result()?;
+        Ok(())
+    }
+}
+
+pub(crate) fn cache_clear_bytes(l2_bytes: usize) -> usize {
+    (l2_bytes * CACHE_CLEAR_L2_MULTIPLIER).max(CACHE_CLEAR_MIN_BYTES)
+}
+
+pub(crate) fn patterned_bf16(len: usize, scale: f32) -> Vec<bf16> {
+    (0..len)
+        .map(|i| bf16::from_f32((((i % 251) as f32) - 125.0) * scale))
+        .collect()
+}