Multi-Layer Annotation: Stacking Scales on One Transcript

Most annotation projects run one scheme at a time: code the transcript for symptoms, ship it, move on. But a transcript is not a single-use resource. The same interview can carry an emotion label, a speech-act label, and a clinical severity rating at once, and each of those layers answers a different research question. Multi-layer annotation is the discipline of stacking those schemes on one text without letting them contaminate each other.

The payoff is efficiency. One pass of annotator effort produces several aligned datasets instead of one, and the raw transcript gets annotated once and reused many times (Pustejovsky & Stubbs, 2012). The cost is a set of design decisions most single-scheme projects never face: how to store overlapping spans, whether the layers are allowed to talk to each other, and how to keep each layer’s reliability honest and separate.

Get those three decisions right and a multi-layer annotation of a clinical corpus becomes far more valuable than the sum of its passes. Get them wrong and you get a tangle of tags no one can score.

What is multi-layer annotation?

Multi-layer annotation is the practice of applying more than one coding scheme or rating scale to the same primary text, keeping each scheme’s labels in a distinct layer instead of merging them into one flat set of tags. Each layer is a complete annotation in its own right: its own categories, its own units, its own reliability. The transcript underneath them all stays fixed.

The idea comes straight from corpus linguistics, where a single text is routinely annotated for part of speech, syntax, coreference, and semantics as separate layers, then merged for analysis (Pustejovsky & Stubbs, 2012). Ide and Suderman make the general case: a graph-based representation can hold “not only independent linguistic annotations, but also sets of merged annotations as a single graph,” which is exactly what you need when several schemes describe the same words (Ide & Suderman, 2007).

For clinical and behavioral transcript work, the layers are often drawn from unrelated traditions. You might combine a dimensional emotion scheme, a pragmatics scheme for what each utterance is doing, and a symptom rating scale. None of those were designed with the others in mind, and that is the point: each captures something the others miss, on one shared piece of evidence.

Stand-off vs inline: how should the layers be stored?

Store multi-layer annotations stand-off, not inline, whenever more than one layer can mark the same region. In inline (embedded) markup, tags are interleaved with the text; in stand-off annotation, each layer is a separate document that refers to the primary text by character offset (ISO 24612:2012). The distinction sounds like plumbing, but it decides whether overlapping layers are even representable.

Inline markup is fine for a single scheme and fails fast for several. Two layers that need to mark overlapping spans force their tags to nest, and nested tags have to stay well-formed, so the moment an emotion span and a symptom span cross without containing each other, the format breaks (Ide & Suderman, 2007). Stand-off sidesteps this entirely: because each layer points into the text rather than wrapping it, two separate annotation documents can lay overlapping hierarchies over the same primary data—the model the Linguistic Annotation Framework standardizes (ISO 24612:2012).

Aspect	Inline (embedded) markup	Stand-off annotation
Where the layers live	Interleaved with the text as tags	Separate documents referencing character offsets (ISO 24612:2012)
Overlapping spans across layers	Hard—tag nesting must stay well-formed	Native—layers may overlap freely (Ide & Suderman, 2007)
Adding or removing a layer	Re-edit the source text	Add or drop a stand-off file; source untouched
Reusing the raw text	Coupled to one scheme	One primary text, many independent layers (Pustejovsky & Stubbs, 2012)
Merging for analysis	Manual reconciliation	Merge as a single graph (GrAF; Ide & Suderman, 2007)

The practical rule: if you can foresee ever adding a second layer, start stand-off. Retrofitting overlap-safe storage after coders have worked inline is far more expensive than choosing it up front.

Independent vs dependent layers: do the schemes interact?

The second design decision is whether your layers are independent or dependent, and it has consequences the storage question does not. Independent layers are coded without reference to one another, so each can be annotated, reused, and scored on its own. Dependent layers condition one layer on another, for example rating sentiment only on spans that a prior layer already tagged as a symptom.

Independence is the safer default because it isolates reliability and maximizes reuse. When an emotion layer and a speech-act layer are coded blind to each other, a disagreement in one cannot leak into the other, and either layer can be handed to a different study untouched.

Dependence buys you something specific in return: it lets you ask questions about interaction, such as whether a particular speech act tends to co-occur with a particular symptom. The price is error propagation. If the upstream layer is wrong on a span, every downstream judgment inherited from that span is built on sand, so the downstream layer’s agreement is conditional on the upstream one, not absolute.

Property	Independent layers	Dependent layers
How each layer is coded	Blind to the other layers	One layer conditioned on another's labels
Worked example	Emotion (Ekman) and speech act (Searle) coded separately	Sentiment coded only on spans tagged as a symptom
Reliability	Scored per layer, in isolation	Downstream IRR is conditional on the upstream layer
Main risk	None from layer ordering	Error propagates from upstream to downstream
Reuse	Any layer can be lifted for another study	Downstream layer is bound to its upstream
Use it when	You want clean, reusable, isolated measures	The question is genuinely about interaction

A useful discipline: keep the layers independent during coding even if you plan to study their interaction later. You can always compute co-occurrence between two independently coded layers after the fact, and doing so keeps each layer’s reliability interpretable. Baking the dependency into the coding step is a one-way door.

A worked example: one utterance, three layers

Here is a synthetic utterance—invented text, no real interview data—annotated with three layers at once. Suppose a subject says:

“Honestly? I promise I’ll take the medication, but nothing is going to help, everything feels pointless.”

Layer one is emotion, using Ekman’s basic emotions, a categorical scheme of six emotions (anger, disgust, fear, happiness, sadness, surprise) applied to spans (Ekman, 1992). A coder tags “everything feels pointless” as sadness.

Layer two is speech act, using Searle’s taxonomy of five illocutionary classes (assertives, directives, commissives, expressives, declarations), which asks what each utterance is doing (Searle, 1976). Here “I promise I’ll take the medication” is a commissive, and “nothing is going to help” is an assertive.

Layer three is clinical severity, using the Montgomery-Åsberg Depression Rating Scale (MADRS), a ten-item clinician rating with each item scored 0 to 6 (Montgomery & Åsberg, 1979). The pessimistic content (“nothing is going to help, everything feels pointless”) is evidence toward the pessimistic-thoughts item, rated for the interview as a whole.

Layer	Scheme	Unit	Label on this utterance	Reliability coefficient
Emotion	Ekman basic emotions (Ekman, 1992)	Span	Sadness on "everything feels pointless"	Kappa or nominal alpha, plus span F1/IoU
Speech act	Searle taxonomy (Searle, 1976)	Span (clause)	Commissive; Assertive	Kappa or nominal alpha, plus span F1/IoU
Clinical severity	MADRS (Montgomery & Åsberg, 1979)	Whole interview	Pessimistic-thoughts item rated on 0–6	Weighted kappa or an ICC

Notice three things. The layers use different units: two are span-level and categorical, one is a whole-interview ordinal rating. They overlap: the sadness span and the pessimistic-thoughts evidence sit on the same words. And they stay independent: the emotion coder does not need the MADRS rating, and vice versa. That last property is what lets you score each layer on its own terms.

How do overlapping spans work across layers?

Overlapping spans are a feature of multi-layer annotation, not a bug, and stand-off storage is what makes them clean. Within a single layer, overlap is usually a codebook question—can two categories mark the same characters?—but across layers, overlap is expected: the whole reason to stack schemes is that they describe the same words from different angles. In the worked example, “everything feels pointless” carries an Ekman sadness tag and contributes MADRS evidence simultaneously, and nothing about that is a conflict.

Because each layer references the primary text by offset rather than wrapping it, two layers can mark the same span, partially overlapping spans, or crossing spans without any well-formedness problem (ISO 24612:2012). This is precisely the case that inline markup cannot handle, and it is why the graph model behind GrAF represents the merged result as a single graph over shared primary data rather than as nested tags (Ide & Suderman, 2007).

Overlap within one layer is a different matter, and it is where you still need a rule. If your span-level coding lets two annotators mark boundaries slightly differently, you are back to a span-agreement problem, which is measured with overlap metrics rather than exact-match kappa—the subject of our guide to agreement for spans with F1 and IoU. Cross-layer overlap needs no such rule; same-layer boundary overlap does.

How do you keep per-layer reliability separate?

Compute inter-rater reliability once per layer, never once for the whole stack. The governing principle, from Hayes and Krippendorff, is that a reliability index must match the level of measurement of the data (Hayes & Krippendorff, 2007). Your layers sit at different levels—categorical emotion, categorical speech act, ordinal severity—so a single pooled coefficient would have to pretend they share a unit and a chance baseline, which they do not.

Match each coefficient to its layer. A categorical layer such as Ekman emotion or Searle speech act takes a chance-corrected kappa or nominal Krippendorff’s alpha, read against a stated benchmark rather than a bare adjective (Landis & Koch, 1977). An ordinal rating such as a MADRS item takes weighted kappa, which credits a near miss more than a far one (Cohen, 1968), or an intraclass correlation for a summed score. Span-level layers need an overlap metric on top of category agreement, because agreeing on the label is not the same as agreeing on the boundary. The full map of which coefficient fits which data lives in our guide to choosing an inter-rater reliability coefficient.

Reporting follows the same rule. Give each layer its own agreement figure, its own coefficient name, and its own benchmark, so a reader can trust the emotion layer even if the severity layer came out weaker. Averaging them into one headline number hides exactly the information a methods reviewer needs, and it lets a strong layer paper over a broken one. When two coders diverge inside a layer, that boundary is where the codebook for that layer needs a sharper rule—an argument we make in full for reading annotator disagreement as signal, not noise.

What does multi-layer annotation demand from your tooling?

Multi-layer annotation needs tooling built for it, not a single-scheme tool used three times. The features that matter are the ability to define several layers with independent tagsets, to show and hide individual layers while coding, to compute agreement per layer, and to export in a format that keeps the layers separable.

WebAnno, a general-purpose linguistic annotation tool, is a clear reference point: it supports multiple annotation layers with freely configurable tagsets, per-project agreement measurement, and an adjudication interface (Yimam et al., 2013). Its designers found something worth heeding—“displaying all annotation layers at the same time is inconvenient for annotators”—so the tool lets each coder show or hide layers individually (Yimam et al., 2013). If you have ever tried to read five overlapping colored spans on one screen, you know why that toggle is not a luxury.

Export format is the last, easily forgotten requirement. If the layers come out flattened into one tag stream, you have lost the separability you paid for. Stand-off export—each layer preserved as its own set of offset-anchored annotations over the shared text—keeps every layer independently analyzable and lets you merge them deliberately later (ISO 24612:2012; Ide & Suderman, 2007). This is also what makes it practical to bring in a model-assisted first pass for one layer, a workflow we look at in multimodal annotation with foundation models, without disturbing the human-coded layers beside it.

Where Tagaroo fits

Tagaroo is a schema-first annotation workspace built to hold several layers on one transcript, so the instrument follows the question instead of the tool. You can define a categorical coding scheme with span selection, an ordinal rating with visible severity anchors, and more than one of each in the same project, then show or hide layers as you work.

Inter-rater reliability is computed per layer, with the coefficient chosen to match each layer’s measurement level rather than defaulted to whatever is easiest. An AI first pass can pre-highlight candidate spans for a layer, but it never issues a finished label and never a diagnosis. If you are turning a published instrument into one of those layers, our walkthrough on going from a rating scale to a codebook covers the operational steps.

On data handling, Tagaroo takes a de-identify-first path: its terms require you to strip direct identifiers before upload, its anonymous trial mode runs in the browser so trial text stays on your machine, and the stack is EU-hosted and GDPR-oriented. It is not a medical device. If your workflow must process raw identifiable transcripts in the cloud, put that question to any vendor—including this one—before uploading; see the privacy policy for specifics.

The practical upshot

Multi-layer annotation turns a transcript from a single-use artifact into a reusable one, but only if you make three decisions deliberately. Store the layers stand-off so overlapping spans are representable and any layer can be added or dropped without touching the source. Keep the layers independent unless the interaction between schemes is the actual thing you are studying. And score each layer’s reliability on its own, with a coefficient matched to its measurement level, because a pooled number across layers means nothing.

If you change one thing, change this: stop reporting a single agreement figure for a stacked transcript, and report one per layer instead. Then build the layered scheme in Tagaroo—an emotion layer, a speech-act layer, a severity layer, whatever your question needs—with the matching reliability coefficient running on each, and let one transcript answer several questions at once.

References

• Pustejovsky, J., & Stubbs, A. (2012). Natural Language Annotation for Machine Learning. O’Reilly Media. Publisher
• Ide, N., & Suderman, K. (2007). GrAF: A Graph-based Format for Linguistic Annotations. In Proceedings of the Linguistic Annotation Workshop, 1–8. Association for Computational Linguistics. ACL Anthology
• International Organization for Standardization. (2012). ISO 24612:2012 Language resource management—Linguistic annotation framework (LAF). ISO
• Yimam, S. M., Gurevych, I., Eckart de Castilho, R., & Biemann, C. (2013). WebAnno: A Flexible, Web-based and Visually Supported System for Distributed Annotations. In Proceedings of the 51st Annual Meeting of the ACL: System Demonstrations, 1–6. ACL Anthology
• Hayes, A. F., & Krippendorff, K. (2007). Answering the Call for a Standard Reliability Measure for Coding Data. Communication Methods and Measures, 1(1), 77–89. doi:10.1080/19312450709336664
• Landis, J. R., & Koch, G. G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33(1), 159–174. doi:10.2307/2529310
• Cohen, J. (1968). Weighted kappa: Nominal scale agreement with provision for scaled disagreement or partial credit. Psychological Bulletin, 70(4), 213–220. doi:10.1037/h0026256
• Krippendorff, K. (2018). Content Analysis: An Introduction to Its Methodology (4th ed.). SAGE Publications.
• Ekman, P. (1992). An argument for basic emotions. Cognition and Emotion, 6(3–4), 169–200. doi:10.1080/02699939208411068
• Searle, J. R. (1976). A classification of illocutionary acts. Language in Society, 5(1), 1–23. doi:10.1017/S0047404500006837
• Montgomery, S. A., & Åsberg, M. (1979). A new depression scale designed to be sensitive to change. British Journal of Psychiatry, 134, 382–389. doi:10.1192/bjp.134.4.382