Speed up the cache-aware streaming RNNT pipeline - #16231

Open

naymaraq wants to merge 16 commits into

mainfrom

dkaramyan/cache-aware-streaming-rtfx

Commits on Sep 8, 2026

asr_streaming_infer: read audio once before the timed loop so RTFx excludes disk I/O
naymaraq
committed
5a9d07e
manifest_io: sort input longest-first so the continuous-batching tail is short
naymaraq
committed
8bd9198
cache_feature_bufferer: shift the whole batch of feature buffers at once
naymaraq
committed
60cee0b
LSTMDropout: unroll the single-timestep eval step with torch.lstm_cell
naymaraq
committed
dd1cc75
Review: bound the audio preloading and drop hardware-specific numbers
naymaraq
committed
0d5f191
rnn: document LSTMDropout.forward for pylint
naymaraq
committed
79d5f53
LSTMDropout: keep the cuDNN path while tracing
naymaraq
committed
f2d36ea
LSTMDropout: keep the cuDNN path under autocast
naymaraq
committed
a6cde05
cache-aware pipeline: compile the encoder layers with torch.compile
naymaraq
committed
c3a3b54
cache-aware pipeline: compile the encoder as one graph with the chunk length pinned
naymaraq
committed
b9174d8
context manager: drop the identity gather and rebuild no slot tensors per step
naymaraq
committed
1d95cc4
cache aware rnnt: one step for a batch mixing last and non-last chunks
naymaraq
committed
ed6e4ff
cache aware feature bufferer: keep the batch batched across the step boundary
naymaraq
committed
db7871d
cache_aware_rnnt_inference_wrapper: black formatting
naymaraq
committed
f865039

Commits on Sep 10, 2026

fix(asr): keep the fused subsampling geometry as ints for torch.compile

naymaraq
and
claude
committed
6dadd33
Merge branch 'main' into dkaramyan/cache-aware-streaming-rtfx
naymaraq
authored
c3bb6a7