Transkripce
Česká verze

Speaker Diarization: How Software Tells Speakers Apart

· 6 min read ·

A recording of a meeting with ten participants looks like one continuous stream of sound with no voice distinction. Speaker diarization is the technique that splits this stream and assigns each utterance to a specific voice. How it works, where it reliably stops working — and what that means for processing multi-speaker recordings.

What Diarization Is and What It Is Not

Diarization (from the English "speaker diarization," with a distant etymological connection to "diary") refers to automatically dividing an audio recording into segments based on who is speaking. The result is a timeline: "Speaker 1 talks from 00:12 to 00:45, Speaker 2 from 00:46 to 01:12..."

Diarization by itself does not transcribe. It creates a time structure with assigned voices, not text. Text is produced only by combining transcription with diarization — and the result then looks like: "[Speaker A] Good morning. [Speaker B] Good morning, how are you? [Speaker A] Fine, thanks."

Without diarization, the transcript of a multi-speaker recording is one continuous stream of text with no indication of who said what. That may suffice for capturing content — but not for quoting, analyzing roles, or structuring meeting records.

Where diarization is used: transcription of meetings and group interviews, academic research interviews, call centers (agent versus customer), legal and administrative records, news or documentary programmes with multiple guests.

How Software Distinguishes Different Voices

Every voice has unique acoustic properties: pitch, tempo, timbre, and the resonance characteristics of the vocal tract. The algorithm extracts and compares these properties — similar to a fingerprint, but for sound.

Speaker Embeddings — an Acoustic Signature

A neural network analyzes each short sound segment (typically 1–3 seconds) and extracts from it a vector of numbers — an embedding. This vector captures the typical spectral properties of the voice in that segment. Embeddings are learned so that segments from the same speaker produce similar vectors, while segments from different speakers produce different ones.

The most widely used architectures are d-vector (LSTM-based), x-vector (TDNN-based), and the newer ECAPA-TDNN. Snyder et al. (2018) showed that x-vectors trained on sufficiently large datasets achieve significantly better reliability than older methods.

Clustering — Grouping Similar Voices

Embeddings from all segments of the recording are clustered. Segments with similar embeddings belong to the same speaker — or so the model assumes. The number of clusters corresponds to the expected number of speakers (either specified by the user or estimated by the algorithm).

Clustering methods include: agglomerative hierarchical clustering (repeatedly merges the most similar clusters), spectral clustering, PLDA (Probabilistic Linear Discriminant Analysis) scoring for pairwise segment comparison.

The algorithm does not know how many speakers are in the recording — it must estimate this itself if the user does not provide the count. The estimate can be wrong, especially when two speakers sound similar or when one speaker noticeably changes their voice.

From Cluster to Identity

Without a database of registered voice samples: the system labels speakers as "Speaker A," "Speaker B" — not by name. The user then assigns real names to the codes.

With a registered voice sample: speaker identification — assigning a cluster to a specific person by comparison with a stored sample. This mode is technically more demanding and requires a pre-recorded sample from each person.

Where Diarization Works Reliably and Where It Fails

The technical principle is elegant. The reality of a group discussion is less favorable.

Favorable Conditions

Diarization works best when:

Under these conditions, diarization works reliably and saves significant time compared to manual labeling.

Typical Errors and Their Causes

Voice overlap: The biggest problem. If two people speak at the same time, the algorithm has no clean embedding for either of them. The result depends on which voice dominates — the second speaker's assignment will likely be incorrect or missing.

Acoustically similar voices: Two men of similar age, accent, and speaking pace — their embeddings overlap. The algorithm may merge them into one cluster and assign portions of the recording incorrectly.

Short utterances: Single-word or single-sentence responses provide too short a segment for a reliable embedding. Brief affirmations like "mm," "yeah," "right" may be assigned arbitrarily.

Noise and reverberation: These distort the spectral properties of the voice. An embedding from a noisy segment may not match an embedding from a quiet segment of the same speaker — and the algorithm will assign them to different clusters.

Variable voice: Emotion, fatigue, whispering, laughter — the same speaker sounds different. The algorithm may split one speaker into multiple clusters.

How Many Speakers Can the System Handle?

Google Speech-to-Text and ElevenLabs Scribe claim support for up to 32 speakers. In practice: reliability drops significantly with the number of speakers, especially in degraded acoustic conditions. For meeting transcripts with 5–8 people, results are usable as a starting point — but they require review.

How to Work with Diarization Results

Diarization saves time but does not replace human review. Typical errors are predictable — and therefore correctable.

What to Always Check

Segment boundaries: At the transition between utterances, the algorithm may assign the end of one sentence to the wrong speaker — especially when speakers alternate rapidly.

Short interjections: "mm," "yeah," "right" — verify that these are assigned correctly. Usually not the top priority, but in a one-on-one interview, an incorrectly assigned brief interjection can confuse the reader.

Overlaps: If simultaneous speech occurred in the recording, check how the segment was processed and whether the result is comprehensible.

Export Formats

The JSON format preserves speakers as metadata on each segment (speaker_label: "speaker_0"). VTT and SRT formats may include speaker information in the header of each subtitle block. For details on export formats see A22.

How to Record for Better Diarization

Each participant on their own microphone or channel dramatically improves results, because each voice arrives without cross-talk from other speakers. A stereo recording with each speaker on their own channel is the ideal foundation for diarization. Speaking in turns and minimizing overlaps — simple organizational measures with significant impact on quality. For a detailed guide to recording preparation see A12.

Conclusion

Speaker diarization is a technology that, under favorable conditions, saves hours of manual voice labeling. In a two-person interview in a quiet room it works reliably. In a group discussion with ten colleagues, the result is indicative — a valuable starting point that requires review.

Realistic expectations are the foundation for correct deployment. Diarization will correctly assign 70–85% of utterances and allow you to correct the remaining portion with far less effort than doing everything manually. For a comprehensive view of meeting transcription — including terminology, recording structure, and recording recommendations — the topic continues in A28.

Sources

  1. Snyder, D., Garcia-Romero, D., Sell, G., Povey, D. & Khudanpur, S. (2018). X-vectors: Robust DNN Embeddings for Speaker Recognition. ICASSP 2018. [doi:10.1109/ICASSP.2018.8461375]
  2. Park, T. et al. (2022). A Review of Speaker Diarization: Recent Advances with Deep Learning. arXiv. [doi:arXiv:2101.09624]
  3. Sell, G. & Garcia-Romero, D. (2014). Speaker Diarization with PLDA i-vector Scoring and Unsupervised Calibration. SLT 2014.
  4. Google Cloud Speech-to-Text — Speaker Diarization. https://cloud.google.com/speech-to-text/docs/diarization
  5. ElevenLabs Scribe — Speaker diarization documentation. https://elevenlabs.io/docs/scribe