Span-Level Agreement: Why Kappa Fails, Use F1 and IoU

Reach for Cohen’s kappa on a span-annotation project and you hit a wall that has nothing to do with your data: kappa needs a fixed list of items and a countable “no” answer, and span annotation gives you neither. When two coders mark stretches of a transcript—a phrase of disorganized speech, a named entity—they are not filling in a pre-drawn grid; they are deciding first where the units are, and only then what to call them. That single difference breaks the arithmetic of chance-corrected agreement and pushes span-level agreement onto a different footing: pairwise F1, and, for how closely two spans line up, intersection over union. This guide gives the formulas, a worked synthetic example, and a clear rule for what counts as a match.

Why doesn’t Cohen’s kappa work for spans?

Cohen’s kappa fails for spans because it is built for a task span annotation is not. Kappa scores a fixed roster of items that every coder rates into the same categories, and it corrects the raw agreement for how often two coders would match by chance. To do that correction it needs the full contingency table, including the cell where both coders said “no.” Span and entity annotation never supplies that cell.

The problem is the negative class. In a categorization task—say, rating each of 40 interview turns as present or absent for a symptom—the items are given and the “no” answers are countable, but in named-entity or span work the items are not given at all. As Hripcsak and Rothschild (2005) note, when annotators mark phrases in a document there is no fixed denominator: they choose the spans, so the number of things that could have been marked but weren’t is unbounded. You cannot count the non-entities in a paragraph any more than you can count the sentences you didn’t write.

Deléger and colleagues (2012) reach the same conclusion from the clinical-NLP trenches: “kappa requires the number of negative cases to be computed, which is unknown in the case of named entities. Named entities are sequences of words, and there is no pre-existing fixed number of items to consider when annotating a text.” Without that count, chance agreement is undefined, and a chance-corrected coefficient has nothing to correct against.

How do you compute span-level agreement with F1?

Span-level agreement is measured by treating agreement as a retrieval problem: pick one annotator as the reference, treat the other as if it were a system trying to reproduce that reference, and compute precision, recall, and their harmonic mean. Precision is the fraction of the second coder’s spans that match the reference; recall is the fraction of the reference’s spans the second coder recovered. F1 = 2·P·R / (P+R) combines them.

The elegant part is symmetry. Swap the roles—call the second coder the reference instead—and precision and recall trade places, but F1 is unchanged, so it does not matter which annotator you label “gold” (Deléger et al., 2012). That is what makes a system-evaluation metric legitimate as an agreement measure between two equals.

There is a deeper reason F1 is the right tool here, not just a convenient one. Hripcsak and Rothschild (2005) proved that “the average F-measure among pairs of experts is numerically identical to the average positive specific agreement among experts,” and that kappa “approaches these measures as the number of negative cases grows large.” In other words, the honest measure of agreement on positives is F1; kappa merely converges to it once you stop being able to count negatives.

Worked example: two coders, five spans

Take a short synthetic subject turn and two coders marking the stretches they read as tangential or derailed speech, the kind of judgment the Thought, Language and Communication (TLC) scale calls for. Coder A marks five spans; Coder B also marks five. Using the MUC/SemEval error categories—correct, partial, missed, spurious—their overlap breaks down like this:

• 3 spans marked identically by both coders (exact matches).
• 1 span that overlaps but with different boundaries: A marked a 40-character stretch, B marked a 30-character stretch sitting inside it.
• 1 span A marked that B missed entirely.
• 1 span B marked that A did not (spurious, relative to A).

Score it exact-match first, where the overlapping pair counts as wrong because the boundaries differ. Precision = 3 / 5 = 0.60, recall = 3 / 5 = 0.60, so F1 = 0.60. Now score it partial-match, giving the overlapping pair half credit: precision = (3 + 0.5) / 5 = 0.70, recall = 0.70, so F1 = 0.70.

The overlapping pair is also where boundary IoU earns its place. B’s 30 characters sit entirely inside A’s 40, so the intersection is 30 and the union is 40: IoU = 30 / 40 = 0.75. Under a common “IoU ≥ 0.5 counts as the same span” rule, that pair is a match; tighten the threshold to 0.8 and it is not. The threshold, like the matching scheme, is a decision you make, not a fact the data hands you.

Exact vs partial match: what counts as a span match?

A span match is whatever you define it to be, and the definition moves the number more than the annotators do. The field’s reference point is SemEval-2013 Task 9, which scores span/entity agreement four ways depending on whether boundaries must line up exactly and whether the entity type must also agree (Segura-Bedmar, Martínez & Herrero-Zazo, 2013). Partial and type schemes award an overlapping-but-imperfect span half credit; strict and exact schemes count it as an outright miss.

Scheme	Boundaries	Entity type	Overlap-only span scores as	Use when
Strict	Must match exactly	Must match	Incorrect (no credit)	Both the span and its label must be right
Exact	Must match exactly	Ignored	Incorrect (no credit)	Only the span boundaries matter
Partial	Overlap is enough	Ignored	Half credit	Approximate spans are acceptable
Type	Some overlap required	Must match	Half credit	The label matters; boundaries can be loose

The practical rule: pick the scheme that matches your downstream use, then report the scheme with every number. Reporting “F1 = 0.82” without saying whether it is strict or partial is like reporting a temperature without the unit. A de-identification pipeline that redacts a whole phrase can live with partial match; a task that extracts an exact dose string cannot.

How does boundary IoU measure span overlap?

Boundary IoU measures how tightly two spans coincide by dividing the length they share by the total length they cover: IoU = |A∩B| / |A∪B|, where the sets are the two spans’ character or token offsets. It runs from 0 (no overlap) to 1 (identical spans), and it is the one-dimensional cousin of the overlap metric computer vision uses on regions.

That kinship is exact, not loose. Marking a span in text is choosing a start and end on a line; marking a box or mask in an image is choosing a region in a plane; both are unitizing a continuum, and both are scored by intersection over union. Object detection accepts a predicted box when its IoU with the ground truth clears a threshold such as 0.5, and span evaluation can accept a coder’s span the same way. If you want the full treatment of overlap metrics and where they mislead, our guides to Dice vs IoU vs Hausdorff distance and image-segmentation agreement with Dice, IoU, and STAPLE work through the two-dimensional case in detail.

IoU is also what “partial match” quietly rests on. When a scheme says two spans are “similar enough,” IoU is the natural way to make enough precise: report the distribution of pairwise IoU across matched spans, and you have said something about boundary discipline that a single F1 hides.

Token-level vs span-level agreement

There is a tempting shortcut: chop the text into tokens, label each token inside-a-span or outside-a-span, and run kappa on that fixed grid of tokens. Now you have a countable set of items, so kappa is defined. The catch is that this token-level kappa answers a question you did not ask and inflates itself while doing so.

Two failures stack up. First, coders annotate sequences, not tokens, so scoring token by token throws away whether the two agreed on the same span as a unit (Deléger et al., 2012). Second, the non-entity tokens swamp everything. In a 120-token turn where 6 tokens fall inside marked spans and 114 do not, both coders say “outside” almost everywhere, chance agreement is nearly total, and kappa is computed on a wildly imbalanced table—the same class-imbalance trap that produces the kappa paradox on categorical data.

Span-level (entity-level) scoring is the honest default: match whole spans under a stated scheme, compute pairwise F1, and report boundary IoU if boundary precision matters. Token-level F1 can still be a useful secondary diagnostic for how coders disagree, but it is not a substitute for the span-level number.

Is F1 really an agreement measure? Limitations and the chance-corrected option

F1’s honest limitation is the one thing kappa was invented to fix: F1 is not chance-corrected. Two coders who both mark spans generously will share overlaps partly by luck, and F1 counts that luck as agreement. For rare, sharply-bounded entities the inflation is small; for dense, fuzzy spans it is not, and a high F1 can flatter a scheme that two careful coders would still argue over.

If you genuinely need chance correction on a span task, it exists—it is just harder. Krippendorff (1995) built a reliability coefficient for exactly this: “the identification of units for analysis within essentially continuous phenomena,” where coders choose each unit’s location, length, and label. This unitizing alpha models the continuum and corrects for chance where ordinary kappa cannot. The trade-off is interpretability. Artstein and Poesio (2008), surveying agreement across computational linguistics, note that alpha-like coefficients “may be more appropriate for many corpus annotation tasks—but that their use makes the interpretation of the value of the coefficient even harder.”

So the defensible reporting recipe is layered. Report pairwise F1 under a named matching scheme as the headline; add the exact-vs-partial gap to show how boundary-sensitive the task is; add boundary IoU where edges matter; and reach for Krippendorff’s alpha in its unitizing form only when a reviewer or protocol demands a chance-corrected figure. For choosing among coefficients on categorical tasks, our guide to which IRR coefficient to use covers where kappa still belongs.

Which measure should you use for span agreement?

Match the measure to the shape of the task, and state your choices out loud.

• Report pairwise F1 as the headline span-level agreement number, treating each annotator in turn as the reference. It is symmetric and equals positive specific agreement (Hripcsak & Rothschild, 2005).
• Name your matching scheme every time. Strict, exact, partial, or type (Segura-Bedmar, Martínez & Herrero-Zazo, 2013); an F1 without its scheme is not interpretable.
• Add boundary IoU when edges carry meaning. The same overlap metric as image segmentation, thresholded to decide matches.
• Don’t run raw Cohen’s kappa on spans. With no fixed items and an unbounded negative class it is undefined; the token-level workaround just re-derives F1 (Deléger et al., 2012).
• Escalate to Krippendorff’s unitizing alpha only when you specifically need chance correction and can absorb the harder interpretation (Krippendorff, 1995; Artstein & Poesio, 2008).

This is the split that recurs across annotation work: where a thing sits is a span-overlap problem scored by F1 and IoU, while what to call a pre-set unit is a categorization problem scored by Cohen’s kappa. Coding speech-act boundaries against the Searle taxonomy or disorganized-speech spans against the TLC scale lives on the first side of that line; rating a whole interview turn lives on the second. Tagaroo computes span-level agreement as coders work—pairwise F1 with your matching scheme, boundary IoU where it matters—so the number in your methods section is one you can defend. If your spans are landing in documents rather than transcripts, our guide to document and PDF annotation covers the same measurement in that setting.

References

• Hripcsak, G., & Rothschild, A. S. (2005). Agreement, the F-Measure, and Reliability in Information Retrieval. Journal of the American Medical Informatics Association, 12(3), 296–298. doi.org/10.1197/jamia.M1733
• Deléger, L., Li, Q., Lingren, T., Kaiser, M., Molnár, K., Stoutenborough, L., Kouril, M., Marsolo, K., & Solti, I. (2012). Building Gold Standard Corpora for Medical Natural Language Processing Tasks. AMIA Annual Symposium Proceedings, 2012, 144–153. pmc.ncbi.nlm.nih.gov/articles/PMC3540456
• Segura-Bedmar, I., Martínez, P., & Herrero-Zazo, M. (2013). SemEval-2013 Task 9: Extraction of Drug-Drug Interactions from Biomedical Texts (DDIExtraction 2013). Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013), 341–350. aclanthology.org/S13-2056
• Artstein, R., & Poesio, M. (2008). Survey Article: Inter-Coder Agreement for Computational Linguistics. Computational Linguistics, 34(4), 555–596. doi.org/10.1162/coli.07-034-R2
• Krippendorff, K. (1995). On the Reliability of Unitizing Continuous Data. Sociological Methodology, 25, 47–76. doi.org/10.2307/271061