Nemotron 3 vs pyannote: Diarization Error Rate 16.0% → 9.9%

Nemotron 3 lowers the diarization error rate from 16.0% to 9.9%

The diarization error rate (DER) adds up three kinds of error, measured in speech time: speech given to the wrong person, speech the model missed, and noise taken for speech. It is the standard measure of speaker diarization. The table puts it next to the problems users notice most, and next to the speed and memory of both models on a Mac and an iPhone.

Green, here and in every table and chart below, marks the better value where the difference is beyond the margin of error: for accuracy, a 95% interval that excludes zero; for speed and memory, a gap far larger than the spread between runs. Error rates: lower is better. The accuracy rows come from 122 public recordings (25 hours, English and Chinese). Speed and memory are for both models running through Core ML, with Nemotron 3 on the Neural Engine; Mac times are processing only, and iPhone times run from the start of detection to the result.

The drop does not come from a few recordings. The chart below shows the change on each of the 122 recordings, from the largest drop to the largest rise:

Each bar is one recording and shows how many points its error rate changed with Nemotron 3: below zero it fell, above zero it rose.

Nemotron 3 is worse in three cases: recordings with more than eight speakers, recordings with a single speaker, and paragraph breaks in the middle of a sentence.

Speaker confusion nearly disappears: 4.5% → 1.0%

Of the three kinds of error, speech given to the wrong person is the one a reader notices first: the words are correct, but the name is wrong. With Nemotron 3 it nearly disappears, and missed speech falls as well.

Average over the 122 recordings, in percent of speech time; both bars in each row start at zero. The drops in wrong-speaker and missed speech are beyond the margin of error. The difference in noise taken for speech is within it, so that bar is light grey. Values are rounded, so they may not add up exactly to the total.

The right number of speakers in 80% of recordings, up from 52%

Finding too few speakers means that two voices were merged into one, so two people's turns come out under a single label. This was the most common mistake of pyannote community-1, and Nemotron 3 makes it far less often.

Number of speakers each model found, compared with the real number, on the 122 recordings. pyannote community-1 mostly found too few. Nemotron 3 is right more often, and when it is wrong, it finds too many as often as too few.

Better on meetings, broadcasts and recordings with two to eight speakers

Recordings differ a lot in difficulty: meetings recorded from across the room are the hardest, broadcasts the easiest. Split by recording type, language and number of speakers, every group with at least ten recordings has a lower error rate; for the one-speaker group, the difference is within the margin of error. The three recordings with nine to twelve speakers got worse, but they are too few to test.

Both bars in each group start at zero: grey is pyannote community-1, green is Nemotron 3, and the number at the end of each bar is the average error rate. Green marks a drop beyond the margin of error, light grey a difference within it, amber a rise. The 6 earnings calls count in the totals but are too few for a group of their own.

Cleaning up a transcript: manual fixes per 10 minutes fall from 28.9 to 18.9

In the end, the error rate turns into work: the number of merges, splits and speaker changes you need to make the speaker view right. We counted the smallest number of such fixes on 4 two-person recordings that have a word-by-word reference (2 Dutch, 1 Chinese, 1 Finnish). Another error readers notice is one person's sentence split between two speakers. On 12 English meetings recorded from across the room, both models rarely do this, and the difference is within the margin of error.

The manual-fix count comes from only 4 recordings, too few for a statistical test or a general claim. A split sentence is a sentence spoken by one person whose text went into two speakers' paragraphs.

Nemotron 3 is faster and needs less than a seventh of the memory

On a device, speed and memory matter as much as accuracy. The table shows processing time and memory for both models running through Core ML, with Nemotron 3 on the Neural Engine. On every device we tested, Nemotron 3 is faster and uses far less memory, which matters most on older iPhones with 4 GB of memory.

Median of several runs (pyannote community-1 ran once on the iPhone 15), all on the same 55-minute earnings call with 10 speakers. Both models run through Core ML, Nemotron 3 on the Neural Engine. Mac times are processing only, without audio decoding or model loading. iPhone times run from the start of detection to the result and include model loading and, for pyannote community-1, the post-processing it shipped with. The phones were plugged in over USB, and each run started with the phone cool.

The first time Core ML runs a model on the Neural Engine, it compiles the model for that device: about 7 seconds on a Mac and about a minute on an iPhone 15. After that, loading the model takes a fraction of a second.

Where pyannote community-1 is still better

• More than eight speakers. Nemotron 3 labels at most eight people; on our 3 recordings with nine to twelve speakers, it did worse than pyannote community-1 on 2.
• A single speaker. On 14 single-speaker recordings, Nemotron 3 found extra speakers in 5; pyannote community-1 in 1.
• Breaks in the middle of a sentence. In 12 English meetings, 25.8% of paragraph breaks fell mid-sentence, against 14.6% with pyannote community-1.

Tips for getting the most out of Nemotron 3 in Whisper Notes

• Set the number of speakers when you know it. On the 108 recordings with two or more speakers, telling Nemotron 3 the number corrected the speaker count on 15 and made it wrong on none, with no clear change in the error rate (+0.16 points, within the margin).
• If only one person speaks, skip speaker detection. That avoids the occasional second "speaker".
• With more than eight speakers, treat the labels as a first draft. Merge and rename speakers in the speaker view where needed.

Setting the number of speakers from Not right? in the Mac speaker view

Speaker detection works the same on iPhone and Mac, entirely on your device. For how to use it in the app, see Whisper Notes Now Identifies Speakers.

How they work: pyannote cuts and clusters, Nemotron 3 numbers speakers in the order they first speak

pyannote community-1 is a three-step pipeline. It cuts the audio into 10-second windows and finds who speaks in each one (up to 3 people per window, up to 2 at the same moment). It then extracts a voiceprint, or embedding, for each speech segment. Finally, VBx clustering groups similar voiceprints into speakers. Speaker numbers inside a window only hold for that window, so the total number of speakers is set only in the last step.

Nemotron 3 Diarization is a single model: a 31-layer Transformer encoder that scores eight speaker tracks at every moment. Each track is scored on its own, so when several people talk at once, several tracks are active. It follows NVIDIA's Sortformer approach and numbers the tracks by when each person first speaks: the first person to speak is always on track 1, so the model never has to guess which slot a voice belongs in. In the streaming setup described on its model card, a speaker cache kept in order of arrival remembers earlier speakers, so a later segment can be matched to the same number.

Each design has its limit. pyannote community-1 has no fixed speaker limit but separates at most two people at the same moment. Nemotron 3 can label several people talking at once but has eight tracks, so it finds at most eight speakers.

The diagrams are simplified. pyannote community-1: about 21 MB in Whisper Notes, CC BY 4.0 (model card); segmentation model by Plaquet and Bredin (Interspeech 2023), embeddings from WeSpeaker, clustering with VBx (Landini et al., 2022). Nemotron 3 Diarization: 97 MB (8-bit Core ML conversion by Fluid Inference), about 100 million parameters, OpenMDW 1.1 (model card); method in the Sortformer and Streaming Sortformer papers.

How we tested

• Recordings. 122 public recordings, 25 hours, in English and Chinese, with 1 to 12 speakers each: meetings recorded from across the room, broadcasts, earnings calls, and single-speaker recordings joined from separate clips. Also 5 two-person recordings in English, Dutch, Chinese and Finnish, 4 of them with a word-by-word reference.
• Datasets. AMI, AISHELL-4, AliMeeting, VoxConverse, Earnings-21, LibriSpeech, IFADV and SKN, plus our own recordings. Only datasets whose licenses allow publishing results; no recordings from our users.
• Metric. Diarization error rate, scored in 10 ms frames with no collar, overlapping speech included, averaged per recording.
• Statistics. 95% intervals from 10,000 bootstrap resamples of whole recordings; no intervals for groups with fewer than ten recordings.
• Devices. A Mac with an M5 chip, an iPhone 15 and an iPhone 12 mini; accuracy was computed on the Mac.
• Pre-registration. The test plan was frozen on 2 November 2026, and the two-person split on 4 November, before any model had run on those clips.

Try it

Whisper Notes on iPhone and Mac now uses Nemotron 3 for speaker detection. If you have questions, write to support@whispernotes.app.

Download for iOS Download for macOS