Audio Annotation: From Transcription to Labeled Events

Audio annotation is where a recording stops being a waveform and starts being data you can train on, search, or measure. The trap is treating it as one job, “get me a transcript,” when it is really a stack of distinct labeling tasks, each with its own output, its own error metric, and its own cost.

A verbatim transcript answers what was said. It says nothing about who said it, when a cough or a long silence broke the flow, or whether the voice was flat and slow. Those are separate layers, and picking the wrong set is how a project overspends on one while starving the layer the model actually needs. This guide lays the layers out in order, with the datasets and metrics that anchor each.

What is audio annotation?

Audio annotation is the practice of attaching structured labels to a sound recording so a machine can learn from it or a researcher can measure it. The labels range from full text of the speech, to speaker-tagged time segments, to timestamped marks on individual sounds, to descriptions of tone and affect. Each answers a different question about the same audio, which is why serious projects treat it as several jobs rather than one.

The reason the distinction matters is cost and fit. A transcript and a prosody tag look similar in a tool, but they demand different annotators, different guidelines, and different quality checks. Speech-recognition corpora such as LibriSpeech aligned roughly 1,000 hours of read audiobook speech to text (Panayotov et al., 2015), while a meeting corpus like AMI layered transcription, dialogue acts, and head-and-gaze annotation over about 100 hours of multi-party recordings (Carletta, 2007). Same medium, very different labels, because the questions differed.

The audio annotation types at a glance

The core audio annotation types are transcription, speaker diarization, segment and event labeling, clip classification, and prosody or paralinguistic tagging. They form a ladder from the literal words toward the way those words were delivered, and each rung produces a different output and is scored with a different metric. Read a row to understand one type; read the last two columns to decide when to reach for it.

Annotation type	What you produce	Time structure	Typical quality metric	Best for
Transcription	Verbatim words as text (optionally word-timed)	Utterance- or word-aligned	Word Error Rate (WER)	Turning speech into searchable, trainable text
Speaker diarization	'Who spoke when' as speaker-labeled segments	Continuous time segments	Diarization Error Rate (DER)	Separating voices in meetings, interviews, calls
Segment / event labeling	A class attached to a start–end span	Explicit time spans	Overlap + inter-rater agreement	Marking where a sound or behavior occurs
Clip / turn classification	One label per clip or speaking turn	Whole clip, no fine timing	Accuracy / F1 + κ	Sorting clips: emotion, intent, topic
Prosody / paralinguistic	Pitch, stress, rate, pause, and affect labels	Aligned to spans or syllables	Inter-rater agreement (κ / α)	Capturing how something is said

This table is the whole argument on one screen: the layers move from literal content toward delivery, and cost tends to rise with how fine-grained the timing and judgment become. The discipline is to annotate only the layers your downstream task consumes. The sections below unpack each, and for the same “types and cost” framing applied to pictures, see our guide to image annotation types.

What is transcription, and why is it only the first layer?

Transcription converts spoken audio into written words, and it is the foundation layer because almost everything else attaches to it. Its standard metric is Word Error Rate (WER): the count of substitutions, deletions, and insertions divided by the number of reference words. WER is why “just transcribe it” hides real difficulty, because the ceiling is not 100%.

The number that reframes the whole layer comes from a human baseline. When Microsoft measured professional transcribers on the Switchboard conversational benchmark, they made errors at about 5.9% WER, rising to 11.3% on the messier CallHome calls (Xiong et al., 2016). Two careful humans transcribing the same call will disagree on filler words, overlaps, and where one word ends. So the practical target is not a flawless transcript but a consistent one, which means writing down the rules for contractions, false starts, numbers, and unintelligible speech before anyone starts.

Transcription also comes in flavors that change the annotation contract. Verbatim keeps every “um,” repetition, and false start; clean or intelligent transcription silently repairs them. For an emotion or clinical study the disfluencies are the signal, so verbatim is mandatory; for a searchable meeting archive, clean reads better. Deciding this up front is not a formatting nicety, it is a labeling decision that ripples into every layer stacked on top.

What is speaker diarization?

Speaker diarization answers “who spoke when” by cutting the timeline into segments and assigning each to a speaker, without necessarily knowing any speaker’s identity. It is what turns one undifferentiated audio stream into an interviewer turn, a subject turn, and a stretch of overlap. For any recording with more than one voice, diarization is the layer that makes a transcript usable, because attribution is half the meaning of a dialogue.

Its metric is Diarization Error Rate (DER), introduced for the NIST Rich Transcription evaluations. DER is the fraction of reference speaker time scored wrong, and it decomposes into three parts: missed speech (real speech the system left unlabeled), false alarm (labeled speech where there was none), and speaker error (speech mapped to the wrong person after an optimal speaker-to-speaker alignment) (Anguera et al., 2012). Reporting the three components separately is more useful than the single number, because missed speech and speaker confusion call for different fixes.

Diarization is hardest exactly where real data lives: crosstalk, laughter, and far-field microphones. The CHiME-5 challenge captured this on purpose by recording natural dinner-party conversations across several microphone arrays in real homes, and the overlapping, distant speech made both recognition and speaker attribution far harder than clean studio audio (Barker et al., 2018). If your recordings are noisy or multi-party, budget for diarization review as its own pass, not a free by-product of transcription.

What is timestamped event and segment labeling?

Segment and event labeling attaches a class to a specific start–end span of audio, marking where something happens rather than transcribing words. This is how you tag a cough, a laugh, a three-second silence, a phone ringing, or a stretch of agitated speech. The output is a list of time-bounded, labeled spans, and it is the audio analogue of drawing a box around a region: you are localizing an event in time instead of in pixels.

The reference dataset for sound events is AudioSet, which built a hierarchical ontology of 632 audio event classes and labeled over two million 10-second clips against it (Gemmeke et al., 2017). That scale exists because event labels are how machines learn to recognize non-speech and paralinguistic sounds, from a doorbell to a sigh. For human-driven span work, tools like ELAN were built precisely for time-accurate, multi-layer annotation of audio and video, letting annotators stack several tiers of labeled intervals on one timeline (Wittenburg et al., 2006); Praat plays a similar role for phonetic segmentation.

The quality question for spans is boundary agreement. Two annotators will rarely start and end a “long pause” at the same millisecond, so event work needs an overlap tolerance and an agreement metric, much as image masks do. The pattern of “coarse label first, precise span only where it matters” transfers directly; we lay out that timing-and-agreement logic for pictures in image annotation types.

What are prosody and emotion annotations?

Prosody and emotion annotations label how speech sounds rather than what it says: pitch movement, loudness, speech rate, stress, and the length and placement of pauses, plus higher-level affect judgments. This is the paralinguistic layer, and it is where audio carries information text simply cannot. The words “I’m fine” transcribe identically whether spoken briskly or in a flat, halting monotone, and only a prosody or emotion label preserves the difference.

For affect, teams usually pick a compact, defined scheme so annotators agree. Ekman’s six basic emotions—anger, disgust, fear, happiness, sadness, and surprise—are a widely used discrete set for tagging emotional content (Ekman, 1992), and you can explore that scheme in Tagaroo’s Ekman basic-emotions scale. The trade is the same one every taxonomy faces: fewer categories buy higher agreement, more categories buy nuance at the cost of rater disagreement.

Prosody annotation matters most in clinical and research speech. A systematic review of automated psychiatric assessment from speech found that acoustic and timing features—speech rate, pausing, pitch variability—recur as candidate markers across depression, schizophrenia, and other conditions, while cautioning that the evidence is still maturing and not diagnostic on its own (Low et al., 2020). Structured span schemes turn that into annotatable work: instruments like the TLC scale for thought, language, and communication give operational definitions for disorganized-speech phenomena an annotator can highlight and label (Andreasen, 1986), and a scale like SANS alogia targets poverty of speech directly.

A worked example: stacking the layers on one snippet

Layers are easiest to see on a single utterance. Here is a short, entirely synthetic interview snippet in Tagaroo’s plain transcript format (no real recording or person):

I: Can you tell me how your sleep has been this week?
S: I don't... I don't really sleep. I just lie there, like, staring.

Now the same six seconds, annotated across the stack. Notice that each layer adds information the layer above could not carry:

Annotation layer	Label applied to the snippet above
Transcription (verbatim)	Words kept exactly, false start and filler retained: “I don’t… I don’t really sleep. I just lie there, like, staring.”
Speaker diarization	00:14.2–00:16.0 = Interviewer; 00:16.4–00:22.1 = Subject
Event / segment	span [00:16.9–00:18.2] = long_pause; point [00:20.6] = filler(“like”)
Turn classification	Subject-turn affect = sadness (Ekman)
Prosody / paralinguistic	Reduced speech rate; flat pitch contour; two pauses over 1.0 s

The transcript alone would let a reader mistake this for neutral content. The stacked annotation records the hesitation, the flat delivery, and the pausing that a clinician or an emotion model would actually care about. That is the argument for thinking in layers: the value often lives in the layers people skip.

How is audio annotation quality measured?

Audio annotation quality is measured per layer, because each layer has its own failure mode. Transcription uses WER; diarization uses DER; classification uses accuracy or F1; and span and prosody labels use overlap plus an inter-rater agreement coefficient. There is no single “audio accuracy” number, and reporting one is usually a sign the layers have been muddled together.

For the categorical layers—turn emotion, event class, prosodic label—chance-corrected agreement is the honest measure, not raw percent agreement. Cohen’s κ and related coefficients discount the agreement you would expect by luck, which matters because a rare event (a genuine long pause, a moment of anger) inflates raw agreement while telling you little. We work through why in Cohen’s kappa and inter-rater reliability. The upstream lever, as in every modality, is the guideline: ambiguous span rules and undefined emotion categories produce disagreement no metric can rescue, which is why the codebook comes before the coding, a point we make at length in annotation guidelines that work.

How does audio annotation compare to text and image annotation?

Audio annotation shares its logic with text and image labeling but adds a time axis that changes the mechanics. Like image annotation, it ranges from coarse whole-item labels to precise localized ones, and the precise end costs more. Like text annotation, its hardest problems are definitional: what counts as a “pause,” where a turn boundary sits, which of six emotions a flat sentence conveys. The extra ingredient is synchronization, since every span label has to line up against a moving timeline, and human annotators disagree on boundaries by design.

The overlap with clinical text work is the tightest, because a transcript is where audio and text annotation meet. Once speech becomes a diarized, verbatim transcript, the downstream coding—emotion, thought-and-language phenomena, communication behaviors—is text annotation with timestamps attached. That handoff is the subject of our guide to clinical transcript annotation tools, and it is exactly the seam where Tagaroo does its work.

Where does Tagaroo fit?

Tagaroo is not an audio-labeling platform and does not compute waveforms, run speech recognition, or draw pitch contours; for those layers, use a speech toolkit or a dedicated audio tool. Where Tagaroo is useful is the layer that begins once you have a transcript: applying a defined coding scheme to spans of text consistently, with inter-rater reliability computed as annotators work. Diarization and prosody get you a rich transcript; the interpretation of that transcript is structured text annotation, and that is the part we build for.

That interpretation layer is where curated schemes make the granularity trade explicit. A compact affect set like Ekman’s basic emotions buys speed and high agreement; a clinical instrument like the TLC scale gives operational, highlightable definitions for disorganized-speech phenomena (Andreasen, 1986). Tagaroo turns either into a guided span-annotation workflow, which is the same discipline a serious audio program needs once the sound has become text.

How to choose your audio annotation layers

Choosing audio annotation layers comes down to one question asked in order: what does the downstream task actually consume? Work up from the cheapest layer and stop at the last one your model or analysis genuinely uses.

• Need searchable or trainable text → transcription, with verbatim-versus-clean decided up front.
• More than one voice → add speaker diarization, and review it as its own pass on noisy or multi-party audio.
• Need to know when a sound or behavior occurs → add timestamped event/segment labels with a boundary tolerance.
• Sorting clips or turns into categories → clip/turn classification with chance-corrected agreement.
• The delivery carries the signal → prosody/emotion tags, using a defined scheme such as Ekman’s emotions.

The practical upshot: audio annotation is a layered stack, not a single deliverable, and the winning move is to annotate only the layers your question needs, write the guideline before the first label, and measure each layer with its own metric. When your audio has become a transcript and the real work is applying a coding scheme, start with a curated scheme like Ekman’s basic emotions or the TLC thought-and-language scale in Tagaroo, and bring the inter-rater reliability discipline with you.

References

• Panayotov V, Chen G, Povey D, Khudanpur S. Librispeech: an ASR corpus based on public domain audio books. ICASSP 2015, pp. 5206–5210. DOI · dataset
• Carletta J. Unleashing the killer corpus: experiences in creating the multi-everything AMI Meeting Corpus. Language Resources and Evaluation. 2007;41(2):181–190. DOI
• Gemmeke JF, Ellis DPW, Freedman D, Jansen A, Lawrence W, Moore RC, Plakal M, Ritter M. Audio Set: An ontology and human-labeled dataset for audio events. ICASSP 2017, pp. 776–780. DOI
• Barker J, Watanabe S, Vincent E, Trmal J. The fifth ‘CHiME’ Speech Separation and Recognition Challenge: Dataset, task and baselines. Interspeech 2018. arXiv:1803.10609
• Anguera X, Bozonnet S, Evans N, Fredouille C, Friedland G, Vinyals O. Speaker Diarization: A Review of Recent Research. IEEE Transactions on Audio, Speech, and Language Processing. 2012;20(2):356–370. DOI
• Wittenburg P, Brugman H, Russel A, Klassmann A, Sloetjes H. ELAN: a Professional Framework for Multimodality Research. LREC 2006, pp. 1556–1559. ACL Anthology
• Xiong W, Droppo J, Huang X, Seide F, Seltzer M, Stolcke A, Yu D, Zweig G. Achieving Human Parity in Conversational Speech Recognition. Microsoft Research MSR-TR-2016-71, 2016. arXiv:1610.05256
• Low DM, Bentley KH, Ghosh SS. Automated assessment of psychiatric disorders using speech: A systematic review. Laryngoscope Investigative Otolaryngology. 2020;5(1):96–116. DOI
• Andreasen NC. The Scale for the Assessment of Thought, Language, and Communication (TLC). Schizophrenia Bulletin. 1986;12(3):473–482. DOI
• Ekman P. An argument for basic emotions. Cognition and Emotion. 1992;6(3–4):169–200. DOI

Educational content, not clinical or engineering advice, and not a diagnostic instrument. Speech-based measures of mental state are an active research area, not validated diagnostics (Low et al., 2020). Disclosure: Tagaroo makes annotation software for structured text and coding schemes; it is not an audio-labeling platform, and this guide names dedicated audio tools where they fit. Handle interview and clinical recordings under a lawful basis and de-identify or gain consent; see our privacy policy.

If your annotation problem is the text and coding layer rather than the waveform, Tagaroo turns schemes like Ekman’s basic emotions and the TLC scale into guided, reliability-tracked workflows.