Skip to content

feat(tts): vendor Perth watermarking in EasyMagpie codec stage - #16208

Open
yhayarannvidia wants to merge 5 commits into
NVIDIA-NeMo:mainfrom
yhayarannvidia:dev/yhayaran/easymagpie_vllm_omni/perth_watermarking
Open

feat(tts): vendor Perth watermarking in EasyMagpie codec stage#16208
yhayarannvidia wants to merge 5 commits into
NVIDIA-NeMo:mainfrom
yhayarannvidia:dev/yhayaran/easymagpie_vllm_omni/perth_watermarking

Conversation

@yhayarannvidia

Copy link
Copy Markdown

Important

The Update branch button must only be pressed in very rare occassions.
An outdated branch is never blocking the merge of a PR.
Please reach out to the automation team before pressing that button.

What does this PR do ?

Watermark EasyMagpie codec PCM with Perth inside the vLLM-Omni package, with no extra PYTHONPATH module.

Collection: [Note which collection this PR will affect]

Changelog

  • Add easymagpie_vllm_omni.watermark and call it from the native codec stage after decode.
  • Enable by default; disable with NEMOTRON_TTS_PERTH_WATERMARK=0. Startup fails if Perth cannot load.
  • Add resemble-perth plus librosa/torchaudio deps to EasyMagpie requirements.txt.
  • Document the env flag; add unit tests and disable watermarking in the codec trim test.

Usage

# default: watermarking on
bash tools/easymagpie_vllm_omni/scripts/run_server.sh ./converted_model 8091
# A/B: disable
NEMOTRON_TTS_PERTH_WATERMARK=0 bash tools/easymagpie_vllm_omni/scripts/run_server.sh ./converted_model 8091

# GitHub Actions CI

The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.

Trusted PRs run automatically through copy-pr-bot. For an untrusted PR, a maintainer can trigger CI by commenting
`/ok to test <head-sha>`; repeat this after a new push if the PR remains untrusted.

# Before your PR is "Ready for review"

**Pre checks**:

- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA-NeMo/Speech/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:

- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.

## Who can review?

Anyone in the community is free to review the PR once the checks have passed.
[Contributor guidelines](https://github.com/NVIDIA-NeMo/Speech/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information

- Related to # (issue)

@copy-pr-bot

copy-pr-bot Bot commented Sep 2, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

numpy>=1.26
pooch==1.8.2
PyYAML>=6.0
resemble-perth==1.0.1

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does perth pull in all of these requirements? If so, can we make perth optional and create another section in requirements? Lastly, let's not pin all of perth's requirements. It can install it's own dependencies.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sure

@svcnvidia-nemo-ci svcnvidia-nemo-ci added the waiting-on-customer Waiting on the original author to respond label Sep 2, 2026
@yhayarannvidia
yhayarannvidia requested a review from a team as a code owner September 4, 2026 04:39
@vklimkov-nvidia

Copy link
Copy Markdown
Member

some generic comments to PR:

  • need benchmark of what is the overhead introduced by watermarking
  • need to explicitly mention that watermarking model works at 32khz. so we would either upsample (from 22khz) of dont do resampling (32khz in the future)
  • biggest problem - you do watermarking independently on separate chunks. currently easymp configured to have chunks of 6 or 8 frames (480 or 640ms) when running codec. chunk size is configurable and I am afraid current watermark impl is very brittle. Internally watermark implementation extracts 2048 FFT with 320-sample hop. then applies 5 convolutions with kernel_size=7. Thats receptive field of 150ms on the left and right. Depending on codec/chunk_size/watermarking - you may get clicks and discontinuities (boundary effects). I don't know how much it affects right now, but theoretically for artifact free streaming watermarking you'd need 150ms delay and 150ms history for each chunk.
  • less of a problem, but worth checking - watermark is applied only in regions with high energy. for 480ms you can't eval threshold reliably and you'd watermark low-energy chunks:
score[t] = normalized_magnitude[:, t].sum()
threshold = 0.05 * score.max()
mask[t] = score[t] > threshold
  • any reason why watermark encoder is global variable that is lazily initialized in codec? why not simply create it in codec constructor?

…ons at chunk edges

This update introduces the `_taper_watermark_delta` function, which applies a tapering effect to the watermark perturbations at the edges of independently processed chunks. This enhancement aims to preserve the codec waveform at both edges, preventing audible seams during the concatenation of streaming chunks. The `watermark_waveforms` function has been updated to utilize this new tapering method, improving the overall audio quality of the watermarked outputs.
…encies

This commit introduces several improvements to the watermarking functionality, including the addition of a new test for preserving chunk edge samples and updates to existing tests for better accuracy. The Dockerfile and requirements files have been updated to include a new `requirements-perth.txt` for optional dependencies, ensuring that the serving loader uses only necessary packages. The README has also been updated to clarify the use of dependencies and their impact on watermarking.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

community-request TTS waiting-on-customer Waiting on the original author to respond

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants