annotation methodology
Evidence Mode vs Instance Mode: Evidence-Based Annotation
Evidence mode rates the whole transcript once and cites its supporting spans; instance mode tags each occurrence. How the choice reshapes reliability math.
Ask two coders to annotate the same interview for a symptom and they can hand you results that do not even have the same shape. One returns a single severity number with a few sentences underlined as justification. The other returns a list of six highlighted utterances, each stamped with a label.
Both are doing evidence-based annotation—grounding a label in the actual text—but they have made a choice, usually without noticing, that decides the unit they measure, the statistic that can score their agreement, and the shape of the data a model later trains on. Tagaroo names that choice explicitly: every scale is either evidence mode or instance mode. This piece is about what separates them and how to pick.
What is evidence-based annotation?
Evidence-based annotation is any coding scheme in which a label is bound to a specific piece of the source instead of standing alone as a verdict. The label always answers “on what basis?” with a pointer into the text. That pointer is what makes a score checkable by a second reader, and it is the common ground beneath both modes—the disagreement is only about what the pointer attaches to.
Evidence mode rates a construct once over the whole transcript and highlights the passages that support the rating. The output is a score plus its evidence: item 3 is a 4, because of these lines. The spans are supporting citations for a judgment that lives at the document level.
Instance mode marks each occurrence of the phenomenon as its own annotation. The output is a set of located, labeled spans: here is a derailment, here is another, here is a speech act. The span is not evidence for a label—the span is the labeled thing. Count them and you have a frequency; map them and you have positions.
The distinction is not cosmetic, and it is upstream of a related one. Whether an instrument is a rating scale or a coding scheme largely determines which mode fits, but the mode is the concrete decision an annotator acts on: attach the label to the document, or to the span.
Evidence mode vs instance mode: what actually changes
The two modes diverge on six axes that matter more the further downstream you go. The table below is the whole argument in one view; the sections after it justify each row.
| Dimension | Evidence mode | Instance mode |
|---|---|---|
| Coding unit (Krippendorff, 2018) | The whole transcript—one rating per item | Each occurrence—an utterance or span |
| What you output | A score + the spans that justify it | A set of located, labeled instances |
| Where the label lives | On the document; spans are evidence | On the span; the span is the phenomenon |
| Reliability question | Do raters agree on the score? | Same spans? And then, same labels? |
| Reliability statistic | Weighted kappa / ICC on the rating | Span-detection F1 or IoU, then label agreement |
| Best for | Dimensional severity over the interview | Enumerable, discrete discourse events |
| Example Tagaroo scales | MADRS, HAM-D, PHQ-9, GAD-7, YMRS | TLC, SANS alogia, Searle speech acts, MITI |
Why is the unit of analysis the real fork?
The unit of analysis is the real fork because everything else follows from it. Krippendorff’s framework separates the sampling unit (what you selected to analyze) from the recording or coding unit (what actually receives a category), and warns that reliability is a property of the coding unit, not of the material in general (Krippendorff, 2018). Evidence mode and instance mode simply choose different coding units for the same sampling unit.
In evidence mode, the coding unit is the whole transcript. Each item is scored exactly once, and the highlighted spans are context—they explain the score without being scored themselves. There is one datum per item per interview.
In instance mode, the coding unit is the individual occurrence. This is what behavioral researchers call event coding: you record each instance of a behavior as it appears, so the number of data points is itself an outcome (Bakeman & Gottman, 1997). Two interviews of equal length can yield three instances or thirty, and that difference is signal, not noise. Choose the unit and you have already chosen whether “how many” is a question your data can answer.
How does the reliability math differ between the modes?
The reliability math differs because the modes give an agreement coefficient different things to compare. When the unit is fixed, agreement is a clean, well-behaved quantity; when the unit is itself uncertain, agreement has to be earned twice.
Evidence mode hands you a fixed unit, so reliability is ordinary category or score agreement: two raters each produce one rating for the item, and you score how well those ratings match with weighted Cohen’s kappa (for ordinal anchors) or an intraclass correlation (for the continuous total). This is the comfortable case that Cohen’s kappa and its relatives were built for, with the usual interpretive bands (Landis & Koch, 1977). One rating in, one rating out, one number back.
Instance mode has no fixed unit, so reliability is a two-stage question. First, did the coders even mark the same spans? That is a detection problem, scored with span-level F1 or intersection-over-union, not with kappa—because kappa needs a shared denominator of items to agree or disagree on, and unmatched spans do not supply one. Only once you have paired the spans can you ask the second question: did the coders give the matched spans the same label?
This is precisely the situation Artstein and Poesio flag when they distinguish agreement on marking from agreement on labeling, and note that once annotators must find the units themselves, the neat chance-corrected story gets more complicated (Artstein & Poesio, 2008). Krippendorff’s unitizing alpha exists for exactly this reason. The practical consequence: an instance-mode project that reports only a labeling kappa has quietly assumed away its harder half.
The same transcript, annotated both ways
Nothing makes the split concrete like running one transcript through both modes. Here is a short synthetic exchange that happens to carry both depressive content and disorganized speech, so each mode has something to grab.
Interviewer: Can you tell me how the last couple of weeks have been?
Subject: Heavy. I wake up and there’s this weight on me, and—you know how a garden goes in autumn, the leaves, my mother kept a garden, she’d be out at six every morning, and anyway. I haven’t wanted to see anyone. Nothing feels worth it. Sorry, what did you ask?
In evidence mode, you rate the depression construct over the whole passage and cite the lines that justify each rating. One rating per item; the spans are the receipts.
| MADRS item (0–6) | Rating (once, whole passage) | Evidence span cited |
|---|---|---|
| Reported Sadness | 4 | "Heavy" / "there's this weight on me" |
| Inability to Feel | 3 | "Nothing feels worth it" |
| Lassitude / withdrawal | 3 | "I haven't wanted to see anyone" |
In instance mode, you ignore the running severity judgment and instead tag each occurrence of a thought-language phenomenon as its own located span.
| Instance # | Located span (synthetic) | TLC label |
|---|---|---|
| 1 | "you know how a garden goes in autumn … she'd be out at six every morning" | Derailment |
| 2 | "and anyway." | Loss of Goal |
| 3 | "Sorry, what did you ask?" | Loss of Goal |
Look at what the two tables are. The first is one column of scores with evidence attached; the second is a list whose length is meaningful. Now imagine a second coder: in evidence mode they hand back three more scores and you compare item to item.
In instance mode they might mark two derailments where you marked one, so before you can compare labels you have to decide whether their span and yours are the same event—the detection stage from the previous section, made visible. The MADRS is evidence mode by design, the TLC is instance mode by design, and the difference in these tables is not a stylistic preference but the scale’s evidence_mode setting doing its job.
Rate the depression items in the tool below to see how evidence-mode item scores roll up into a defensible total.
MADRS score simulator
Rate each item on the 0-6 MADRS anchors: 0 = symptom absent, 6 = most severe. Defined descriptions sit at 0, 2, 4, and 6; the odd values (1, 3, 5) mark intermediate states.
- 1.Apparent sadnessObserved dejection in voice, expression, and posture
- 2.Reported sadnessThe subject's own account of low mood
- 3.Inner tensionEdginess, ill-defined discomfort, dread, or panic
- 4.Reduced sleepReduced duration or depth vs. the subject's normal
- 5.Reduced appetiteLoss of appetite or of the pleasure of eating
- 6.Concentration difficultiesTrouble collecting or sustaining thoughts
- 7.LassitudeDifficulty starting and carrying out activities
- 8.Inability to feelReduced interest; loss of the capacity to feel
- 9.Pessimistic thoughtsGuilt, self-reproach, inferiority, remorse
- 10.Suicidal thoughtsLife not worth living; wishes for death
Total score
0 / 60
Severity band
Normal / symptom-free
Remission (<=10)
Yes
Item 10 flag
None
Severity bands follow Snaith et al. (1986); the remission target of a total of 10 or below follows Hawley et al. (2002). This tool illustrates how the MADRS is scored; it is an educational aid, not a diagnostic instrument, and a score is a measure of severity, not a diagnosis. Adjust the items above (or load the example) to see the score update.
When should you use each mode?
Match the mode to the shape of the construct, not to habit. If the phenomenon is a countable event you want to locate every time it appears—a speech act, a disorganized utterance, a motivational-interviewing behavior—use instance mode, because the count and the positions are results you care about. If the construct is a severity or quality judgment made once over the whole sample, use evidence mode, because there is one thing to rate and the spans exist to justify it.
Tagaroo’s own library shows how cleanly the split falls out in practice. This is a first-hand count from the Scale Library, not a survey of the field.
The evidence-mode scales are the dimensional clinician ratings—MADRS, HAM-D, PHQ-9, GAD-7, YMRS—where each item is one severity call over the interview. The instance-mode scales are the discourse and behavioral coding schemes, where the point is to tag each occurrence: the TLC for formal thought disorder and the SANS alogia subscale for impoverished speech are canonical examples, each item a feature an annotator highlights and labels wherever it occurs.
The split is not always at the level of a whole scale. In Tagaroo’s data the mode is a property of the item, so a scale can mix modes—a couple of the psychiatric rating scales set most items to evidence mode but flip one or two conspicuous items to instance mode where a specific behavior is better counted than rated. If you are building your own scale, decide the mode item by item, and see rating scale vs coding scheme for the upstream instrument choice.
What does each mode give a machine-learning pipeline?
Each mode produces a distinctly more useful training signal than a bare label, in a shape that matches a different model. A grounded label tells a model not just what but where, and that “where” is supervision most datasets throw away.
Evidence mode yields a document-level label paired with a rationale—the exact structure the ERASER benchmark uses to test whether a model’s stated reasons actually align with human-marked evidence rather than with spurious cues (DeYoung et al., 2020). Rationale supervision is not a new idea in principle: annotator rationales have been shown to improve text classifiers when folded into training (Zaidan & Eisner, 2008). Evidence-mode output drops straight into that setup.
Instance mode yields span-level supervision: labeled, located segments that are the native input for sequence-labeling and detection models. The count and the boundaries carry information a document label cannot. In other words, the two modes do not just organize your human coding—they hand two different model families the exact form of ground truth each one wants.
Where the two-mode split breaks down
The dichotomy is a useful default, not a law, and it frays at three edges. First, some constructs are genuinely mixed: a clinician-rated scale may need a running severity judgment and a count of discrete events, which is why per-item mode exists rather than one setting per scale. Forcing such a scale into a single mode loses information either way.
Second, evidence mode can only ground what the text contains. Observational items—psychomotor slowing, affect read from a face rather than from words—live in what the rater saw, not in the transcript, so a text span cannot fully justify them. That is a limit of transcript annotation, not of the mode.
Third, grounding buys provenance, not validity. A perfectly located instance of the wrong category, or a beautifully cited score from an ill-chosen scale, is still wrong—it is just wrong in a way you can now inspect. Evidence-based annotation makes a label auditable; choosing the right label and the right scale is a separate discipline, and one worth pairing with auditable scale scoring so the total you report can defend itself.
Making evidence-based annotation a deliberate choice
The practical upshot: evidence mode and instance mode are two answers to a single question—what does the label attach to?—and the answer sets your unit of analysis, your reliability statistic, and the shape of your training data before a single span is highlighted. Rate the whole transcript and cite the evidence when the construct is one severity judgment; tag every occurrence when it is a countable event. Make the choice on purpose, per construct, and the rest of the pipeline stops fighting you.
References
- Artstein, R., & Poesio, M. (2008). Inter-coder agreement for computational linguistics. Computational Linguistics, 34(4), 555–596. doi:10.1162/coli.07-034-R2
- Bakeman, R., & Gottman, J. M. (1997). Observing interaction: An introduction to sequential analysis (2nd ed.). Cambridge University Press.
- DeYoung, J., Jain, S., Rajani, N. F., Lehman, E., Xiong, C., Socher, R., & Wallace, B. C. (2020). ERASER: A benchmark to evaluate rationalized NLP models. ACL 2020. arXiv:1911.03429
- Krippendorff, K. (2018). Content analysis: An introduction to its methodology (4th ed.). SAGE Publications.
- Landis, J. R., & Koch, G. G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33(1), 159–174. doi:10.2307/2529310
- Montgomery, S. A., & Åsberg, M. (1979). A new depression scale designed to be sensitive to change. British Journal of Psychiatry, 134, 382–389. doi:10.1192/bjp.134.4.382
- Andreasen, N. C. (1984). The Scale for the Assessment of Negative Symptoms (SANS). University of Iowa.
- Andreasen, N. C. (1986). The Scale for the Assessment of Thought, Language, and Communication (TLC). Schizophrenia Bulletin, 12(3), 473–482. doi:10.1093/schbul/12.3.473
- Zaidan, O. F., & Eisner, J. (2008). Modeling annotators: A generative approach to learning from annotator rationales. EMNLP 2008.
If you code symptoms or discourse from interviews, Tagaroo builds both modes into the workflow: evidence-mode scales like the MADRS bind each item rating to its transcript span, while instance-mode scales like the TLC let you tag every occurrence as a located instance—with inter-rater reliability computed the right way for each mode as your coders work.
Frequently asked questions
- What is evidence-based annotation?
- Evidence-based annotation is any coding scheme in which a label is tied to a specific piece of the source text rather than left as a free-floating judgment. In Tagaroo it comes in two modes. Evidence mode rates a construct once over the whole transcript and highlights the spans that justify the rating; instance mode marks each occurrence of the phenomenon as its own located, labeled span. The two modes attach the label to different units of analysis (Krippendorff, 2018), which is why they behave differently downstream.
- What is the difference between evidence mode and instance mode annotation?
- The difference is what carries the label. In evidence mode the coding unit is the whole document: you produce one rating per item and cite the supporting passages as evidence, so the score lives on the transcript and the spans back it up. In instance mode the coding unit is the utterance or span: each occurrence is itself an annotation, so the label lives on the span. Evidence mode fits dimensional severity rated over an interview; instance mode fits enumerable, discrete events in discourse (Bakeman & Gottman, 1997).
- Which reliability statistic goes with each annotation mode?
- Evidence mode has a fixed unit, so reliability is agreement on one rating: weighted Cohen's kappa or an intraclass correlation on the item score. Instance mode has no fixed unit, so reliability is a two-stage question, whether coders marked the same spans (span-detection F1 or intersection-over-union) and then whether they agreed on the labels. That unitizing-plus-labeling problem is exactly the case Artstein and Poesio (2008) separate from ordinary category agreement.
- When should I use instance mode instead of evidence mode?
- Use instance mode when the phenomenon is a countable event you want to locate every time it appears, such as a speech act, a thought-disorder utterance, or a motivational-interviewing behavior; the count and the positions are part of the result. Use evidence mode when the construct is a severity or quality judgment made once over the whole sample, such as a depression item, where the spans exist to justify the score rather than to be counted (Bakeman & Gottman, 1997).
- Does evidence-based annotation improve machine-learning training data?
- It can, because a grounded label carries more supervision than a bare one. Evidence mode yields a document label plus a rationale, the format the ERASER benchmark uses to evaluate whether models attend to the right text (DeYoung et al., 2020), and annotator rationales have been shown to improve text classifiers (Zaidan & Eisner, 2008). Instance mode yields span-level supervision suited to sequence-labeling and detection models. Both give a model more to learn from than a category alone.
Put this into practice
Tagaroo turns any rating scale or coding scheme into a guided annotation workflow — with inter-rater reliability computed as you go.