Transkripce
Česká verze

Emotion, Tone, and Sentiment in Speech: What Transcription Cannot Yet Capture

· 7 min read ·

Automatic transcription today achieves accuracy that would have seemed improbable just ten years ago. Yet the text produced by transcription is not a faithful copy of spoken language — it is an abstraction that captures words but not the way they were spoken. Tone, tempo, vocal tension, or a brief hesitation before an answer carry information that does not transfer into the transcribed document. This article explains what exactly transcription loses, how the field has attempted to fill this gap with text-based sentiment analysis, and why this substitute falls short where genuine speaker emotions matter.

What Is Lost When Audio Becomes Text

Transcription is by its nature a reduction. The input is a continuous acoustic signal rich in information; the output is a sequence of words. This reduction is intentional and useful — that is precisely why transcription exists. But along with noise and irrelevant artefacts, components of the speech signal that are not noise are also lost.

Prosody — The Melody and Rhythm of Speech

Prosody — the melody and rhythm of speech — is among the first casualties of transcription. Pitch carries information about the speaker's attitude: rising intonation signals a question or uncertainty, falling intonation signals closure or decisiveness. The tempo of a conversation says something different from the words alone. A sentence delivered quickly, without pauses, comes across differently than the same sentence spoken slowly with emphasis on individual words. In transcribed text, these two versions are indistinguishable.

Pauses and hesitations are another source of lost information. A brief pause before an answer may signal deliberation, reluctance, or uncertainty. Hesitation sounds like "um" or "uh" are either entirely omitted by transcription systems or captured as textual artefacts without context. The transcription does not process them as signals — they are treated as disruptive elements, not data.

Voice Quality as an Indicator of State

Hoarseness, tremor, or an excessively flat monotone delivery are physiological manifestations of a speaker's emotional or physical state. These characteristics simply cannot be read from text. Repairs — interrupted sentences and their corrections — signal cognitive load or careful word-weighing. Transcription either omits them as speech errors or captures them in a way that does not reflect their communicative function.

Irony and Sarcasm as Examples of Text Failure

Irony and sarcasm depend almost entirely on intonation. The sentence "Excellent, that really turned out well" is a positive statement in transcribed text. In a recording where the speaker rises in pitch and quickens their tempo, it means the exact opposite. The transcription does not capture the difference. The same tonal shift operates at the level of emphasis: "Did you REALLY mean that?" and "Did you really mean that?" are identical statements in text — yet they carry different emotional charges and may differ in communicative intent.

Text-Based Sentiment — What Analysis Can and Cannot Do

The field of natural language processing responded to the absence of emotional data in text by developing sentiment analysis tools. These tools work on text and attempt to derive from lexicon and syntax whether a statement is positive, negative, or neutral.

How Text-Based Sentiment Analysis Works

Lexical approaches, such as VADER, assign weights to individual words and derive overall sentiment from their combination. More modern model-based approaches — particularly BERT-based models like RoBERTa — work with whole-sentence context and can capture subtler nuances. For English, these models are well-trained; for other languages, the situation is less favourable. Most available sentiment models were created primarily for English data, and their transfer to other languages requires fine-tuning on a sufficiently large dataset.

The fundamental structural limitation of text-based sentiment analysis, however, is not linguistic — it is epistemic. Text itself does not contain irony. Politeness conventions mean that a statement like "We'll definitely look into that" can mean refusal just as well as genuine interest; the text does not distinguish. And transcription as the basis for sentiment analysis adds yet another risk: a transcription error — a homophonic substitution, a missing negation, a misrecognised word — transfers directly into the analysis result.

Speech Emotion Recognition — Why It Is Not the Same as Transcription

Speech Emotion Recognition (SER) is a discipline that works directly with the audio signal. Unlike transcription, it does not attempt to convert sound into words — it analyses acoustic features of the voice and classifies the speaker's emotional state based on them. The basic representation for SER is mel spectrograms: graphical displays of frequencies over time that capture both tonal characteristics and voice dynamics. Classifiers trained on this representation distinguish basic emotional categories — anger, joy, sadness, fear, neutral state.

SER and ASR are separate pipelines. Standard transcription systems do not integrate SER for practical reasons: SER requires differently structured training data, different architectures, and achieves significantly lower accuracy than modern ASR — especially on natural spontaneous speech. Laboratories report promising results on controlled data, but real conversations are a different category. Emotions blend within them, do not take pure categorical values, and are strongly culturally conditioned.

Where Emotional Content Matters Most

This gap between transcription and the emotional content of speech is not an academic problem — it has direct impacts in several areas of practice.

Customer Contact Centres

In customer contact centres, supervisory systems work with a combination of transcription and voice analysis as two separate layers. A frustrated customer says "fine" with rising irritation — the text captures agreement, the voice captures the opposite. Systems for detecting critical moments in conversations work directly with the audio signal in real time.

Research Interviews and UX Studies

In research interviews and UX studies, a respondent answers "yes, that works well" — but the voice is monotone, the tempo slow, without inflection. These signals of disinterest or polite accommodation are not captured by the transcript. A researcher working only with the transcript misses this context. Practice therefore combines transcription with timestamps and preserving the audio recording for tracing back critical moments — especially in passages where the researcher is uncertain during analysis.

Mental Health Applications — Sensitive Territory

The most sensitive area is mental health applications. Voice biomarkers of depression — changes in tempo, pitch, voice variability — are the subject of research with potentially clinical applications. Anxiety manifests in rhythmic changes and hesitation patterns. Transcription does not capture these indicators; and even systems that can capture them enter ethically sensitive territory. Collection and analysis of voice data as indicators of mental state require special consent and appropriate personal data protection.

Where Research Is Heading

Cultural variability of emotional expression is a concrete problem for many languages. The way speakers express frustration or enthusiasm through voice differs across cultures. Training datasets for SER are predominantly English or drawn from other major languages. Direct model transfer without adaptation produces unreliable results — and extensive, correctly annotated emotional voice databases remain scarce for many languages.

False positive emotion detection can be more harmful than the absence of detection. A system that labels a neutral statement as an expression of anger can trigger unnecessary interventions, distort research data, or damage user trust in the technology. This risk asymmetry is the reason caution is warranted.

The future likely lies in multimodal models — systems that process audio and text simultaneously rather than sequentially. Research prototypes combining acoustic features with linguistic context achieve better detection of irony or sarcasm than purely text-based approaches. Production deployment for most languages, however, remains a research laboratory prospect, not current practice.

Conclusion

Transcription is a powerful tool for converting spoken language into searchable and processable text. But the textual representation of speech has a structural limitation: it does not capture how the words were spoken. Tone, tempo, prosody, and voice quality are information carriers that do not transfer into text. Text-based sentiment analysis partially compensates for this deficit, but it works with derived information — with what the speaker said, not how they said it. Anyone who works with transcription and cares about emotional context must know this boundary and account for it when interpreting results.

Sources: