optional tail silence to flush the decoder

[naymaraq]

Important

The Update branch button must only be pressed in very rare occassions.
An outdated branch is never blocking the merge of a PR.
Please reach out to the automation team before pressing that button.

What does this PR do ?

Adds streaming.flush_size_in_secs, an opt-in amount of silence appended to the end of every stream
so a streaming decoder can emit the tokens it is still holding when the audio stops. The default is
0.0, which appends nothing and leaves current behaviour unchanged.

Collection: [ASR]

Changelog

• streaming/framing/mono_stream.py: MonoStream takes flush_size_in_secs and appends that many
  seconds of zeros to the samples in load_audio, before framing. They are appended as signal, not
  padding: n_samples covers them, so the frames carrying them report their full length as valid
  and nothing downstream trims them away.
• streaming/framing/multi_stream.py: threads the value through ContinuousBatchedFrameStreamer and
  ContinuousBatchedRequestStreamer to the MonoStream it builds, so one path serves both pipelines.
• pipelines/cache_aware_rnnt_pipeline.py, pipelines/buffered_rnnt_pipeline.py: read
  cfg.streaming.get("flush_size_in_secs", 0.0) and pass it to the request generator. .get keeps
  existing YAMLs working unchanged.
• conf/asr_streaming_inference/cache_aware_rnnt.yaml, buffered_rnnt.yaml: the new key, default 0.0.

Usage

python examples/asr/asr_streaming_inference/asr_streaming_infer.py \
    --config-path=../conf/asr_streaming_inference/ \
    --config-name=cache_aware_rnnt.yaml \
    audio_file=<manifest.json> \
    output_filename=<preds.jsonl> \
    streaming.flush_size_in_secs=0.48

What the parameter buys when it is turned on

nemotron-speech-streaming-en-0.6b, cache-aware RNNT, WER against flush=0.0:

Results, WER

Best value per row and benchmark in bold.

att_context_size [70,13] — 1.12 s lookahead

flush s	LibriSpeech test-other	earnings22	AMI
0.00	0.0756	0.1924	0.1438
0.08	0.0756	0.1924	0.1439
0.16	0.0758	0.1917	0.1451
0.32	0.0757	0.1906	0.1420
0.48	0.0753	0.1878	0.1381
0.64	0.0756	0.1880	0.1402
0.80	0.0756	0.1862	0.1416
1.04	0.0756	0.1866	0.1382

att_context_size [70,6] — 0.56 s lookahead

flush s	LibriSpeech test-other	earnings22	AMI
0.00	0.0779	0.1932	0.1456
0.08	0.0776	0.1938	0.1442
0.16	0.0779	0.1934	0.1443
0.32	0.0778	0.1918	0.1398
0.48	0.0779	0.1898	0.1386
0.64	0.0776	0.1904	0.1397
0.80	0.0780	0.1898	0.1418
1.04	0.0779	0.1887	0.1387

att_context_size [70,1] — 0.16 s lookahead

flush s	LibriSpeech test-other	earnings22	AMI
0.00	0.0820	0.1980	0.1659
0.08	0.0818	0.1969	0.1598
0.16	0.0814	0.1958	0.1577
0.32	0.0813	0.1946	0.1540
0.48	0.0812	0.1936	0.1525
0.64	0.0811	0.1921	0.1510
0.80	0.0813	0.1916	0.1497
1.04	0.0813	0.1909	0.1496

att_context_size [70,0] — 0.08 s lookahead

flush s	LibriSpeech test-other	earnings22	AMI
0.00	0.0904	0.2042	0.1972
0.08	0.0898	0.1993	0.1908
0.16	0.0889	0.1969	0.1851
0.32	0.0881	0.1943	0.1770
0.48	0.0880	0.1932	0.1745
0.64	0.0878	0.1931	0.1724
0.80	0.0879	0.1931	0.1714
1.04	0.0879	0.1931	0.1702

GitHub Actions CI

The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.

Trusted PRs run automatically through copy-pr-bot. For an untrusted PR, a maintainer can trigger CI by commenting
/ok to test <head-sha>; repeat this after a new push if the PR remains untrusted.

Before your PR is "Ready for review"

Pre checks:

• Make sure you read and followed Contributor guidelines
• Did you write any new necessary tests?
• Did you add or update any necessary documentation?
• Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  • Reviewer: Does the PR have correct import guards for all optional libraries?

PR Type:

• New Feature
• Bugfix
• Documentation

If you haven't finished some of the above items you can still open "Draft" PR.

Who can review?

Anyone in the community is free to review the PR once the checks have passed.
Contributor guidelines contains specific people who can review PRs to various areas.

Additional Information

• Related to # (issue)

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[naymaraq]

/ok to test 4e8ff8d

[Copilot]

🟡 Changes recommended

The new flush path can fail on non-CPU tensors (device mismatch) and lacks basic input validation (e.g., negative values), which can break streaming at runtime.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

This PR introduces an opt-in streaming.flush_size_in_secs setting that appends a configurable amount of trailing silence to each streamed utterance, enabling streaming RNNT decoders to emit any “held” tail tokens when audio stops.

Changes:

• Add flush_size_in_secs plumbing through the streaming framing stack so MonoStream appends trailing zeros as signal.
• Read streaming.flush_size_in_secs from ASR streaming pipeline configs and pass it into the request generator.
• Expose the new config key in the example streaming inference YAMLs (default 0.0 to preserve current behavior).

File summaries

File	Description
nemo/collections/asr/inference/streaming/framing/mono_stream.py	Appends optional trailing silence to loaded audio before framing.
nemo/collections/asr/inference/streaming/framing/multi_stream.py	Threads flush_size_in_secs through continuous batched stream/request generators into MonoStream.
nemo/collections/asr/inference/pipelines/cache_aware_rnnt_pipeline.py	Reads streaming.flush_size_in_secs from config and passes it to the request generator.
nemo/collections/asr/inference/pipelines/buffered_rnnt_pipeline.py	Reads streaming.flush_size_in_secs from config and passes it to the request generator.
examples/asr/conf/asr_streaming_inference/cache_aware_rnnt.yaml	Adds streaming.flush_size_in_secs: 0.0 to the example config.
examples/asr/conf/asr_streaming_inference/buffered_rnnt.yaml	Adds streaming.flush_size_in_secs: 0.0 to the example config.

Review details

Suppressed comments (2)

nemo/collections/asr/inference/streaming/framing/mono_stream.py:70

• The flush tail is created with torch.zeros(...) without setting device, which will fail if self.samples is on GPU (or any non-CPU device). Create the zeros tensor on the same device/dtype as self.samples.

        if self.flush_size > 0:  # appended as signal, not padding, so it is not trimmed downstream
            self.samples = torch.cat([self.samples, torch.zeros(self.flush_size, dtype=self.samples.dtype)])

nemo/collections/asr/inference/streaming/framing/mono_stream.py:71

• New flush behavior is introduced here, but there are no tests in this PR that exercise flush_size_in_secs>0 (e.g., verifying that extra samples are appended and that frame lengths/is_last behave as expected). Adding a focused unit test in tests/collections/asr/inference/test_framing.py would prevent regressions.

        if self.flush_size > 0:  # appended as signal, not padding, so it is not trimmed downstream
            self.samples = torch.cat([self.samples, torch.zeros(self.flush_size, dtype=self.samples.dtype)])
        self.n_samples = len(self.samples)

• Files reviewed: 6/6 changed files
• Comments generated: 2
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[naymaraq]

/ok to test 72b9083

[github-actions]

[🤖]: Hi @naymaraq 👋,

We wanted to let you know that a CICD pipeline for this PR just finished successfully.

So it might be time to merge this PR or get some approvals.

[andrusenkoau]

Approved, thank you!