Context
This continues the ASR/Qwen3-ASR infrastructure optimization work tracked in #229. Thanks for getting that groundwork reviewed and merged; this follow-up focuses on one correctness-safe batch decode improvement plus repeatable benchmarking.
Qwen3-ASR batch decode currently underuses Apple Silicon GPU resources for fixed chunk ASR workloads. Official Qwen3-ASR tooling emphasizes vLLM batch inference, max_inference_batch_size, and FlashAttention-style large-batch efficiency, but the MLX Swift path needs a correctness-safe optimization before enabling true batched decoder forwards by default.
Current results
On repeated 10s chunks with Qwen3-ASR 0.6B 4-bit.
Local benchmark environment: MacBook Pro with an M4 chip, fans set to maximum speed.
- 24 chunks, batch size 1: 3.95s inference, aggregate RTF 0.0165
- 24 chunks, batch size 6: 3.55s inference, aggregate RTF 0.0148
- Speedup: 1.113x
- Output digest identical across rows
Batch size 8 was slower than 6 in local sweeps, so the batch-size cap should be benchmark-driven rather than always using the largest possible value.
Proposed implementation
- Keep the default batch path row-correct by running decoder forwards per row.
- Sync one batched
[B, 1] token tensor per decode step instead of B scalar .item() calls.
- Keep true
[B,1,H] decoder forward behind QWEN3_ASR_EXPERIMENTAL_BATCH_DECODE=1 until row correctness is resolved.
- Bucket experimental true-batch decode by encoder sequence length.
- Add a fixed-chunk benchmark harness and a lightweight Apple GPU power monitor.
Known limitation
The true [B,1,H] experimental path still fails repeated-chunk row correctness at batch size 2: row 0 completes, row 1 truncates. It should stay gated until fixed.
Follow-ups
- Add internal prefill/decode/token-count metrics.
- Use Instruments Metal System Trace to check GPU saturation and CPU scalar sync.
- Investigate MLX batched decoder row divergence.
Related: #229
Implementation PR: #234
Context
This continues the ASR/Qwen3-ASR infrastructure optimization work tracked in #229. Thanks for getting that groundwork reviewed and merged; this follow-up focuses on one correctness-safe batch decode improvement plus repeatable benchmarking.
Qwen3-ASR batch decode currently underuses Apple Silicon GPU resources for fixed chunk ASR workloads. Official Qwen3-ASR tooling emphasizes vLLM batch inference,
max_inference_batch_size, and FlashAttention-style large-batch efficiency, but the MLX Swift path needs a correctness-safe optimization before enabling true batched decoder forwards by default.Current results
On repeated 10s chunks with Qwen3-ASR 0.6B 4-bit.
Local benchmark environment: MacBook Pro with an M4 chip, fans set to maximum speed.
Batch size 8 was slower than 6 in local sweeps, so the batch-size cap should be benchmark-driven rather than always using the largest possible value.
Proposed implementation
[B, 1]token tensor per decode step instead of B scalar.item()calls.[B,1,H]decoder forward behindQWEN3_ASR_EXPERIMENTAL_BATCH_DECODE=1until row correctness is resolved.Known limitation
The true
[B,1,H]experimental path still fails repeated-chunk row correctness at batch size 2: row 0 completes, row 1 truncates. It should stay gated until fixed.Follow-ups
Related: #229
Implementation PR: #234