Skip to content

Latest commit

 

History

History
468 lines (387 loc) · 27.7 KB

nemotron_transcribe.md

File metadata and controls

468 lines (387 loc) · 27.7 KB

NVIDIA Nemotron 3.5 Transcribe

nemotron_transcribe is an experimental community port of NVIDIA Nemotron 3.5 Transcribe 30B-A4B. It accepts complete recordings through the asr task. Instructions select transcription, translation, audio Q&A or language identification. Its CTC head provides source-language word timestamps and speaker labels. This family is separate from the smaller nemotron_asr streaming model.

The port uses stock GGML, framework weight stores, the mel frontend and tokenizer, with a model-local parallel speech/speaker encoder and Nemotron-H decoder. NVIDIA's patched serving kernels, MTP and DFlash acceleration are outside this port.

Build

cmake -S . -B build/nemotron -DAUDIOCPP_MODEL_SET=custom \
  -DAUDIOCPP_MODELS=nemotron_transcribe -DAUDIOCPP_DEPLOYMENT_BUILD=ON \
  -DENGINE_ENABLE_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=native
cmake --build build/nemotron --target audiocpp_cli --parallel 4

Omit CUDA for CPU builds. On Windows, use an x64 Visual Studio developer shell and the .exe suffix. Select the destination GPU architecture when distributing a binary.

Download packages

Community GGUF packages are available for this experimental port. Select one decoder and download the shared mmproj-NVIDIA-Nemotron-3.5-Transcribe-30B-A4B-Encoder-F16.gguf beside it. The encoder is 1,670,410,080 bytes and includes the F32 CTC timestamp head. It is a required auxiliary component, not a standalone F16 version of the 30B model. Speech encoder weights use FP16 by default; the CTC head stays FP32. The mmproj- prefix identifies the audio encoder/projection sidecar and keeps it out of the Hub's standalone-model quantization selector. The updated loader also accepts the original native filename without the prefix, preferring it when both names are present. No decoder or encoder tensor data was changed.

Routed expert profile Decoder file Decoder bytes
IQ4_NL nemotron-transcribe-30b-a4b-experts-IQ4_NL.gguf 20,370,389,728
Q4_K nemotron-transcribe-30b-a4b-experts-Q4_K_S.gguf 21,672,721,184
Q3_K nemotron-transcribe-30b-a4b-experts-Q3_K_S.gguf 17,294,261,024
Q2_K nemotron-transcribe-30b-a4b-experts-Q2_K.gguf 13,946,026,784

These labels describe the routed experts. All four packages use Q8_0 dense decoder linears and preserve other BF16/F32 source types. The encoder stays floating-point. File size is not peak runtime memory. Quantization changes predictions; the number of bits does not establish an accuracy ranking.

The Q3_K and Q4_K expert files use _K_S filename suffixes so the Hugging Face selector recognizes them. These custom mixed profiles do not use the stock whole-model llama.cpp _S recipes. The tensor types and weights are unchanged; the conversion flags remain --profile q3_k and --profile q4_k.

The model-manager entries download the decoder and its shared encoder. Pass the chosen decoder file as the model path, rather than the directory containing multiple GGUFs. Specs, configs, tokenizer and license notices are embedded. The repository also contains readable licenses and a per-tensor format inventory.

Run

build/nemotron/bin/audiocpp_cli --task asr --family nemotron_transcribe \
  --model models/audiocpp/nemotron-transcribe-iq4_nl.gguf \
  --backend cuda --audio recording.wav --text-out transcript.txt

Use 16 kHz mono WAV for matched comparisons. Other PCM rates and channel counts are averaged to mono and resampled inside this model, using the framework's Soxr helper with its linear fallback. The shared CLI/server loaders are unchanged. Matched 16 kHz mono inputs pass through without resampling.

Pass each task option with --request-option key=value:

Task Options
Transcription Default; optionally language=ja
Translation template_name=translation, target_language=English
Audio Q&A template_name=audio_qa, instruct=What is discussed in this recording?
Language identification template_name=language_id
Word timestamps return_timestamps=true
Speaker-tagged transcription and timings diarization=true

For the built-in transcription instructions, language is a source-language hint. Common short and regional tags such as zh, zh-CN and ZH_cn resolve to the same language name; custom names such as Mandarin are passed through. The hint does not force the decoder's output language, and long recordings can still produce translations instead of transcripts.

The built-in speaker instruction requests no punctuation or capitalization, matching the reference's trained speaker prompt. To request punctuation as well as speaker tags, supply an explicit instruct that asks for both.

instruct replaces the template instruction, including its language hint and speaker-tag request. --text supplies the instruction when instruct is absent. Response formatting is preserved. Translation and Q&A templates reject timestamps and diarization because their output need not match the source words.

Generation is greedy by default. max_tokens overrides the checkpoint setting or the native duration budget max(24, ceil(6 * seconds - 1e-9) + 24). Prompt, audio and output must fit nemotron_transcribe.max_context, default 73,728. The custom result records token-limit termination. Sampling options are in the model spec.

Optional --request-option repetition_guard=true stops short token loops or sustained replay using NVIDIA NeMo-Speech.cpp's repetition guard. It leaves sampling unchanged and reports termination: "repetition_detected" in the custom result, rather than claiming EOS. The partial transcript is retained. This limits runaway generation; it does not correct wrong-language output or recover missing speech. The default is false because legitimate repeated speech can trigger the guard. Guard state is local to each generation and is not carried between requests.

The server accepts the same options in JSON at /v1/audio/transcriptions. Use /v1/audio/transcriptions/details for word timestamps and speaker turns. The generic /v1/tasks/run route also returns custom_schema_output, including termination and alignment diagnostics. Multipart WAV uploads support the ordinary transcription fields; use JSON options for task templates, questions and timestamp requests. For example:

{"model":"nemotron","audio":"recording.wav","options":{"template_name":"audio_qa","instruct":"What is discussed in this recording?"}}

Replace nemotron with the configured server model ID. The JSON audio path is local to the server. Invalid request options return a client error, including when the recording is silent. top_p=0 keeps the highest-ranked candidate.

The native silence gate returns empty text only when peak amplitude is below 0.001 and RMS is below 0.00003. It does not trim audio or repair generated text.

Model

Field Value
Family nemotron_transcribe
Tasks asr
Modes offline
Languages auto
Default package nemotron_transcribe_iq4_nl
Input Speech audio
Output Transcript text, with optional word timestamps and speaker turns

“Model/session default” means no explicit override. The runtime uses the checkpoint or session setting, or leaves the optional value unset as described.

Common Options (use directly)

Option Meaning
--model Model file or directory shown in the examples.
--backend Select an available backend supported by this model.
--threads CPU thread count.
--log Enable runtime logging.
--audio Input audio, where required by the selected task.

Request Options (use with --request-option)

Option Type / values Default Description
instruct string Model/session default Audio instruction or question. Replaces the template instruction; defaults to text_input when present. Chat controls and audio locators are rejected.
template_name default, translation, audio_qa, language_id default Instruction template. Audio Q&A requires instruct or text_input. Timestamps and speakers require the default transcription template.
language string auto Source language hint for the default transcription prompt. An explicit instruct replaces this prompt, including its language hint.
target_language string English Target language for the translation template.
system_prompt string Model/session default Optional system message; the trained default is empty.
return_timestamps bool false Run the checkpoint CTC head for source-language word timestamps. Failed streams return no fabricated word rows.
diarization bool false Request speaker-tagged transcription, CTC words and speaker activity turns. An explicit instruct must request speaker tags itself.
max_tokens int Model/session default Maximum generated response tokens. Defaults to generation_config or native duration budget max(24,ceil(6*audio_seconds)+24). Prompt and output must fit max_context. Declared range: minimum 1.
do_sample bool false Sample tokens instead of greedy decoding.
temperature float 1 Positive sampling temperature when do_sample is true.
top_k int 0 Sampling candidate count; zero disables top-k filtering. Declared range: minimum 0.
top_p float 1 Nucleus probability when sampling. Declared range: minimum 0, maximum 1.
repetition_guard bool false Stop runaway token repetition and report repetition_detected termination. May truncate genuine repeated speech; does not repair the transcript.
repetition_penalty float 1 Positive token repetition penalty.
seed int 0 Sampling seed. Declared range: minimum 0.

Session Options (use with --session-option)

MoE defaults: resident banks, pinning off, eight cache slots, four read threads, zero CPU MoE layers; CPU share is zero. Use the shared MoE controls via CLI flags, --session-option, or the server model's session_options. Legacy nemotron_transcribe.disk_experts, .pin_experts, .expert_cache_size, and .disk_expert_read_threads aliases remain accepted. For CPU-layer placement, use the shared moe_cpu_expert_layers key. Supplying both a canonical key and its alias is an error, even with equal values. CPU counts select the first N MoE ordinals, skipping dense-only backbone layers. CPU-selected layers override moe_cpu_share and retain bounded CPU caches in staged mode. Pinning is accepted but inactive without staging.

Option Type / values Default Description
moe_disk_experts bool false Keep routed expert weights in the GGUF/host memory and stage selected experts through a bounded backend cache. Dense weights stay on the selected backend; CPU expert placement is independent.
moe_pin_experts bool false With disk_experts, retain complete native expert banks in pinned host RAM for direct uploads. Requires pinned host-buffer support.
moe_expert_cache_size int 8 Resident expert slots per routed layer with disk_experts. Must cover the model's experts per token and cannot exceed its expert count. Declared range: minimum 1.
moe_disk_expert_read_threads int 4 Host workers for staging GGUF expert ranges when disk_experts is true and pin_experts is false. Parallel reads require OpenMP or Apple dispatch. Declared range: minimum 1, maximum 64.
moe_cpu_expert_layers int 0 First N routed MoE layers on CPU; dense-only layers do not count.
moe_cpu_share float 0 CPU share of eligible staged GPU misses, 0–1; requires moe_disk_experts=true. CPU-selected layers take precedence.
nemotron_transcribe.weight_type native, f32, f16, bf16, q8_0 native Native preserves mixed packaged tensor types. Changing this may greatly increase model memory.
nemotron_transcribe.max_context int 73728 Combined text, audio and output context, bounded by backbone positions. Matches native SALM serving default. Declared range: minimum 1.
nemotron_transcribe.prefill_batch_size int 64 Text/audio rows per decoder prefill graph. Declared range: minimum 1, maximum 2048.
nemotron_transcribe.flash_attention bool true Use flash attention in the speech and speaker encoders.
nemotron_transcribe.encoder_window_size int 500 Core encoder window in 80 ms output frames; the native default is 40 seconds. Declared range: minimum 1, maximum 5000.
nemotron_transcribe.encoder_left_context int 50 Encoder window left context in 80 ms frames. Declared range: minimum 0, maximum 5000.
nemotron_transcribe.encoder_right_context int 50 Encoder window right context in 80 ms frames. Declared range: minimum 0, maximum 5000.
nemotron_transcribe.speaker_cache_size int 188 Native serving speaker cache in 80 ms frames, distinct from training configuration. At least two rows per speaker are required. Declared range: minimum 16, maximum 5000.
nemotron_transcribe.speaker_fifo_size int 40 Recent speaker FIFO size in 80 ms frames. Declared range: minimum 0, maximum 5000.
nemotron_transcribe.speaker_update_period int 300 Native serving speaker-cache update period in 80 ms frames. Declared range: minimum 1, maximum 5000.

Prepare weights

Use the full BF16 checkpoint, not the separately packed NVFP4 release. The initial profile has IQ4_NL routed experts, Q8_0 dense decoder linears, BF16/F32 remaining decoder weights, an F16 speech encoder and an F32 CTC head. Native weight storage preserves this mixture.

The package also embeds the exact trained BF16 Hann window. It does not regenerate that window in FP32. Frontend validation uses the actual HF feature extractor and the checkpoint's trained filterbank.

The wrappers require the pinned NVIDIA reference and its pinned llama.cpp submodule. Build a pristine CPU llama-quantize and pass its path explicitly. The wrapper never applies backend patches or builds a quantizer automatically.

git clone https://github.com/NVIDIA/NeMo-Speech.cpp reference/NeMo-Speech.cpp
git -C reference/NeMo-Speech.cpp checkout 66d50d91b75a406becd95fbaf39d2c9c13e95d7c
git -C reference/NeMo-Speech.cpp submodule update --init --depth 1 llama.cpp
cmake -S reference/NeMo-Speech.cpp/llama.cpp -B build/nemotron-quantizer \
  -DGGML_CUDA=OFF -DLLAMA_BUILD_COMMON=ON -DLLAMA_BUILD_TOOLS=ON \
  -DLLAMA_BUILD_TESTS=OFF -DLLAMA_BUILD_EXAMPLES=OFF -DLLAMA_BUILD_SERVER=OFF
cmake --build build/nemotron-quantizer --target llama-quantize --parallel 4

Create a Python 3.12 environment with uv and install the pinned reference's requirements.txt. CPU PyTorch suffices for conversion. The package wrapper uses gguf==0.19.0 and numpy==1.26.4.

Download model revision a99e33f698257b8b69ca7834f19c73582eb1fa7e, including configuration, tokenizer and license files. The checkpoint is about 67.5 GB. Allow about 160 GB free for source, temporary BF16 decoder and GGUF outputs. Source and output must share a filesystem for hardlink staging; Windows does not need symbolic-link privileges.

uv run --no-project --python .venv-nemotron python \
  tools/community_models/convert_nemotron_transcribe.py \
  --source models/NVIDIA-Nemotron-3.5-Transcribe-30B-A4B \
  --reference reference/NeMo-Speech.cpp \
  --quantizer build/nemotron-quantizer/bin/llama-quantize \
  --output models/native/nemotron-transcribe-iq4_nl.gguf --threads 16

uv run --no-project --python .venv-nemotron python \
  tools/community_models/prepare_nemotron_transcribe.py \
  --source models/NVIDIA-Nemotron-3.5-Transcribe-30B-A4B \
  --decoder models/native/nemotron-transcribe-iq4_nl.gguf \
  --encoder models/native/NVIDIA-Nemotron-3.5-Transcribe-30B-A4B-Encoder-F16.gguf \
  --output models/audiocpp/nemotron-transcribe-iq4_nl.gguf

The package is two GGUFs in one directory. Pass the decoder to audio.cpp; it names its encoder sibling. The decoder embeds its v1 spec, configs, tokenizer and notices. The encoder includes its CTC tokenizer. Checkpoint IDs must match, and packaging checks source identity, config and tokenizer semantics. Keep the rest of the encoder filename unchanged. Native conversion writes the original filename; the runtime accepts that name or the same basename prefixed with mmproj-, as used by the public downloads.

Q4_K, Q3_K and Q2_K expert profiles

The K profiles use genuine Q4_K, Q3_K or Q2_K routed expert tensors. Dense decoder linears use Q8_0; embeddings, output projection and other floating-point tensors retain their source types. They share the F16 speech encoder and F32 CTC head. The names describe the routed banks, rather than every tensor in the package.

The logical widths 2,688 and 1,856 do not fit a 256-element K block. A standard quantization request can therefore fall back to a different type. The converter pads only routed-bank input columns to 2,816 and 2,048 with zeros, and audits the actual output types. The model adds matching zero input activations. Logical output widths, routing, normalization and dense projections remain unchanged. Four-bit K storage can be larger than IQ4_NL because of these padded columns.

These files require this community model's metadata-controlled row-padding support. Stock NeMo-Speech.cpp and llama.cpp inference do not load this layout. The existing unpadded IQ4_NL path remains available.

Build the pristine pinned quantizer above. Export one floating-point decoder and its paired encoder directly from the full checkpoint:

uv run --no-project --python .venv-nemotron python \
  tools/community_models/export_nemotron_transcribe_bf16.py \
  --source models/NVIDIA-Nemotron-3.5-Transcribe-30B-A4B \
  --reference reference/NeMo-Speech.cpp \
  --output models/native/nemotron-transcribe-source-BF16.gguf

The exporter reuses an existing encoder only when its checkpoint identity and CTC component agree. --encoder-only exports/checks that sibling without writing the decoder. Conversion work is retained for inspection. Source tensors use hardlinks, so the source and export destination must share a filesystem.

Create a self-contained package, embedding the spec, configs, tokenizer and exact trained frontend window while streaming one shared padded cache:

uv run --no-project --python .venv-nemotron python \
  tools/community_models/quantize_nemotron_transcribe.py \
  --source-bf16-gguf models/native/nemotron-transcribe-source-BF16.gguf \
  --quantizer build/nemotron-quantizer/bin/llama-quantize \
  --profile q4_k --threads 16 \
  --padded-cache models/native/nemotron-transcribe-row256-package-cache.gguf \
  --package-source models/NVIDIA-Nemotron-3.5-Transcribe-30B-A4B \
  --encoder models/native/NVIDIA-Nemotron-3.5-Transcribe-30B-A4B-Encoder-F16.gguf \
  --output models/audiocpp/nemotron-transcribe-30b-a4b-experts-Q4_K_S.gguf

Repeat with --profile q3_k and q2_k, changing the output filename accordingly. Keep the source, cache and package arguments identical to reuse the cache and encoder. No separate packaging copy is needed. The optional --model-spec selects a spec file; freeze it while creating a set of variants. A cache with different source or package inputs is rejected.

The converter never requantizes an existing quantized model. It checks the pinned quantizer, every tensor's actual type and physical shape, pairing metadata and unchanged floating-point payloads. It retains a quantization receipt and refuses to overwrite existing or partial artifacts. Allow roughly 270 GB for the full source, floating-point export, shared padded cache, three packages and encoder.

The self-generated converter tests do not need the checkpoint or GPU. Set NEMOTRON_TEST_QUANTIZER to the pinned executable to include real quantization and package round-trips, then run:

uv run --no-project --python .venv-nemotron python \
  tests/community_models/nemotron_transcribe/test_quantize.py

In PowerShell, set the executable with $env:NEMOTRON_TEST_QUANTIZER = 'build/nemotron-quantizer/bin/llama-quantize.exe'.

CPU weight offload

moe_disk_experts=true keeps routed expert banks in their native GGUF format in host memory and uploads selected experts into a bounded cache. Dense decoder layers and the speech encoder remain on the selected backend; moe_cpu_expert_layers selects CPU expert execution independently. Offload is disabled by default and requires nemotron_transcribe.weight_type=native.

build/debug/bin/audiocpp_cli --task asr --family nemotron_transcribe \
  --model models/nemotron-transcribe-30b-a4b-experts-Q4_K_S.gguf \
  --backend cuda --audio recording.wav --log \
  --session-option moe_disk_experts=true \
  --session-option moe_expert_cache_size=8 \
  --session-option nemotron_transcribe.prefill_batch_size=1024

The required encoder GGUF must be beside the decoder. The same offload controls apply to Vulkan. Eight resident expert slots per layer is the default; larger caches trade device memory for fewer uploads. The cache must cover the selected experts per token and cannot exceed the checkpoint's expert count. The existing prefill_batch_size control also applies to offload: larger batches amortize expert transfers, use more activation memory and can change numerical rounding.

moe_pin_experts=true optionally retains complete expert banks in pinned host RAM for direct uploads. This consumes host memory approximately equal to the native expert payload and requires backend pinned-buffer support. With pinning disabled, moe_disk_expert_read_threads controls host staging copies (default four); parallel staging requires OpenMP or Apple dispatch. Set it to one in builds without either facility.

Graphs, recurrent state, staging allocations and the bounded expert cache are session-owned. Independent requests reset recurrent state and expert-slot placement. Offloading does not change the router, expert activation or routing weight formulas, but grouped expert matrix multiplication can change floating point rounding and therefore generated text.

Timestamps and speakers

The CTC head is loaded on the first timestamp or diarization request and reused by later requests in the same session. Plain transcription does not load it.

TaskResult.word_timestamps uses sample offsets in the input audio clock. speaker_turns contains 10 ms speaker-activity segments. Custom output nemotron_transcribe.result.v1 includes word times in seconds, speaker column, speaker confidence, original transcript tag, alignment status and termination. speaker_N denotes Sortformer column N. The transcript's spk:N tag can map to a different column. Chinese/Japanese alignment units are characters; Thai uses the reference's combining-mark-aware units.

Unavailable or partially failed alignment retains the transcript and available speaker activity. Failed streams produce no invented word rows. Generic word/turn confidence zero means unavailable; speaker confidence is reported separately.

Long recordings and validation

The input limit is one hour. Bounded encoder windows and a stateful speaker cache/FIFO match the pinned native defaults: 500 core frames plus 50 left/right frames on an 80 ms grid, with cache/FIFO/update 188/40/300. These differ from the HF processor's 30-second chunk setting and the diarizer's training geometry. Session options expose the choices. A native C++ comparison does not establish complete Python long-form parity.

For longer recordings, caller-managed clips matching the source processor's 30-second setting are a useful starting point. Shift each result's times by its source offset. Separate requests reset speaker identities, and cuts can split words. The one-hour acceptance limit describes capacity; it does not guarantee recognition quality for one uninterrupted request.

Local validation used MSVC 14.44, CUDA 12.8.61, an RTX PRO 6000 Blackwell 96 GB, and an i9-13900K. The final package passed 11 CUDA requests covering English and Japanese transcription, translation, Q&A, language identification, two-speaker dialogue, repeated requests, silence and token-limit termination. All nine paired text outputs matched the pinned native reference after removing speaker markers and normalizing whitespace. CPU transcription with timestamps also passed.

Measured request Audio Processing time, model already loaded
CUDA English with timestamps 7.33 s 0.556 s
CUDA distinct two-speaker dialogue with timestamps 49.055 s 1.232 s
CPU English with timestamps, four threads 7.33 s 6.250 s

These are individual requests on a small, self-generated synthetic speech set, with other GPU workloads active. CUDA model loading took 14.35 seconds separately. Loading depends on disk cache and system load; this is not a general cold-load estimate. They are not a held-out accuracy or throughput benchmark. Component checks also cover official HF frontend parity, full real encoder weights, decoder math, actual CTC weights, Unicode alignment and 3,522 tokenizer cases.

Word times are not identical to the native reference. Maximum observed differences were 119 ms in the Japanese clip and 424 ms at one dialogue pause. The largest outlier follows the native C++ frontend's legacy Hann/window-placement/valid-frame policy, which changes a speaker-activity gap. This port follows the checkpoint/HF frontend. Both runtimes also omit one repeated sentence pair in a four-repeat 56.44-second fixture; the distinct 49-second dialogue includes all eight turns. These results do not establish hour-long accuracy. Full-model Metal and Vulkan execution, broad multilingual accuracy and actual hour-long audio remain untested. See the validation record for pinned inputs, settings, commands and observed outputs.

All four downloadable profiles also passed CUDA task checks on the synthetic fixtures, including English/Japanese transcription, translation, audio Q&A, language identification, speaker-tagged dialogue, silence, token-limit reporting and repeated requests. The K converter's six self-generated tests passed with the pristine pinned quantizer, and the C++ padding test checks floating-point parity. These checks establish the exercised routes, not broad accuracy. CTC can attach a later turn's leading word to earlier speaker activity even when the transcript is complete. Treat precise word times and speaker boundaries as experimental.

A follow-up CPU server audit passed 24 HTTP checks, including malformed requests, multipart upload, 48 kHz stereo input and unload/reload. Focused safety tests also passed on Windows and WSL GCC. Storage overrides are priced from tensor metadata before uploads; the server's existing opt-in memory guard can reject an expanded model before allocation. These checks do not establish full-model accuracy at every storage type.

Weights retain OpenMDW-1.1 and component notices. Runtime alignment adaptations and Unicode/numerical attributions are in the model's THIRD_PARTY_NOTICES.md and LICENSE.apache-2.0.

MoE engine policies

The routed backbone accepts independent placement and cache-eviction policies through moe_policy and stage-scoped moe_policy_params.placement.* / moe_policy_params.eviction.*. See shared controls and policy parameters. Omitted policies preserve the model's existing residency and CPU-layer defaults. Active policies require staged experts; shared/dense branches and model state remain on their existing paths.