Nvidia releases Nemotron-3-Diarization, a 100M speaker model with 0.32 second latency

Nvidia has released Nemotron-3-Diarization, an open-weights model with 100 million parameters that works out who speaks when in a recording, for up to eight speakers, in either streaming or offline mode. Streaming runs at an input-buffer latency of as little as 0.32 seconds, and the license is the OpenMDW License Agreement v1.1, which Nvidia states covers commercial and non-commercial use.

The model takes single-channel 16 kHz audio in WAV, FLAC, OPUS or MP3, turns it into a mel-spectrogram and runs it through a 31-layer transformer encoder with rotary positional embeddings, returning per-speaker activity probabilities at 10 ms resolution in a tensor of shape [T, 8]. Streaming works through the Sortformer design, which pairs an Arrival-Order Speaker Cache holding the identities of speakers heard earlier in the conversation with a FIFO queue of recent frames. Four presets trade delay against accuracy, running from 30.4 seconds for offline processing down through 1.04 and 0.64 seconds to the 0.32-second minimum Nvidia recommends.

Voice Arena Research put the model first among 12 diarization systems on its Diarization-Bench v1 leaderboard, scoring 14.7% diarization error rate at a 0 ms collar across 139 conversations and 22 hours of English audio, about 4.6 points ahead of second-placed DiariZen. Diarization error rate combines missed speech, false alarms and speaker confusion into one figure, so lower is better. Nvidia’s own measurements at the 1.04-second setting give 13.18% on DIHARD III and 7.70% on NOTSOFAR1 MHM, against 19.60% and 22.12% for the four-speaker Streaming Sortformer v2.1 it replaces.

Local inference runs through NeMo-Speech.cpp, a native C++ runtime with CPU, Metal and Vulkan backends that loads Nvidia speech checkpoints converted to GGUF and lists this model among the ones it supports. Argmax has put the model into Pro SDK 3, its on-device speech kit for Mac and iPhone, which can be tried through the Argmax Playground app on TestFlight. The company compared six diarization systems for the release.

Nemotron 3 Diarization achieved the lowest error rate (DER) across all 6 systems.

Alongside the C++ runtime, the checkpoint loads in the NeMo framework and in Hugging Face Transformers through AutoModelForAudioFrameClassification, and a browser demo is on Hugging Face. The PyTorch path needs an Nvidia Ampere, Ada Lovelace, Hopper or Blackwell GPU on Linux at BF16 precision, where Nvidia reports a real-time factor of 15,113 at the offline setting and 865 at 1.04 seconds on an RTX PRO 5000 at batch size 32. Nvidia names recordings with more than eight speakers, far-field microphones, noise and reverberation as conditions that raise the error rate.