Span-level agreement calculator

Span annotation breaks chance-corrected agreement, and not for a fixable reason. Cohen's kappa scores a fixed roster of items that every coder rates into the same categories, and to correct for chance it needs the cell where both coders said "no". Span work never supplies that cell, because the coders decide where the units are before deciding what to call them.

The missing negative class

In a categorisation task — rating each of forty interview turns as present or absent for a symptom — the items are given and the negatives are countable. In span or named-entity work the items are not given at all. As Hripcsak and Rothschild (2005) put it, when annotators mark phrases in a document there is no fixed denominator: they choose the spans, so the number of things that could have been marked but were not is unbounded. You cannot count the non-entities in a paragraph any more than you can count the sentences you did not write.

The standard replacement is pairwise F1. Treat one annotator as the reference, compute F1 = 2·P·R / (P+R), and because F1 is symmetric the value is the same whichever annotator you call gold (Deléger et al., 2012) — which is precisely the property an agreement measure needs. Our full write-up of span-level agreement works through the arithmetic; this tool runs it.

The matching rule moves the number more than the annotators do

A span match is whatever you define it to be, and the field's reference point is SemEval-2013 Task 9, which scores four ways depending on whether boundaries must line up exactly and whether the label must also agree. On the worked example this page loads, the same two sets of spans score 0.60 under strict and exact, 0.70 under partial, and 0.80 under type. Nothing about the annotation changed.

That is why the scheme belongs in every reported figure. "F1 = 0.82" without it is like a temperature without the unit. A de-identification pipeline that redacts a whole phrase can live with partial matching; a task extracting an exact dose string cannot.

A correction we owe our own readers

One detail here is easy to get wrong, and we did. Under the type scheme, an overlapping span whose label matches earns full credit, not half. The half-credit category exists only in the partial scheme. An earlier version of our explainer said type awarded 0.5, which made it look harsher than it is; this tool follows the reference implementation, and the post has been corrected. Type is the most forgiving of the four schemes on boundaries while staying strict about labels.

Boundary IoU, and why it is the same idea as image overlap

Boundary IoU divides the length two spans share by the total length they cover: IoU = |A∩B| / |A∪B|. It runs from 0 to 1, and it is what "partial match" quietly rests on — when a scheme says two spans are similar enough, IoU is the natural way to make enough precise.

The kinship with image segmentation is exact rather than metaphorical. Marking a span is choosing a start and end on a line; marking a mask is choosing a region in a plane; both are unitising a continuum, and both are scored by intersection over union. Object detection accepts a predicted box when IoU clears a threshold, and span evaluation can accept a coder's span the same way — which the threshold slider above does. Our Dice ↔ IoU converter handles the two-dimensional case, including the Dice identity and why the same metric misleads on small structures.

The token-level shortcut, and why it flatters itself

There is a tempting workaround: chop the text into tokens, label each one inside-a-span or outside, and run kappa on that fixed grid. Now the items are countable and kappa is defined. The catch is that tokenisation invents the negative class the task lacks — every whitespace-separated word that nobody marked becomes an item both coders agreed on. On a long document that class dominates, so agreement looks near-perfect for the same reason kappa breaks on pixels, and token-level kappa converges to F1 as the non-entity class grows (Hripcsak & Rothschild, 2005). It is a useful diagnostic, not a substitute.

If your labels really are categorical over a fixed item set, kappa is the right tool and our inter-rater reliability calculator computes the whole family with confidence intervals. Deciding between the two is the subject of our guide to choosing an agreement coefficient. Span-level coding of the TLC scale's signs is the case where this page applies: the boundaries of a stretch of disorganised speech are a judgement, and two competent coders routinely disagree about them by a clause.

How the alignment works, and why not greedily

Before anything can be scored, the two annotators' spans have to be matched one-to-one. This tool pairs identical boundaries first — those are the same span, and no optimisation should trade one away — then aligns the remainder with a dynamic program that maximises the number of overlapping pairs, and total IoU as a tiebreak.

The obvious alternative, taking the highest-IoU pair repeatedly, is genuinely worse rather than merely inelegant. Given reference spans [0,10) and [10,30) against system spans [5,28) and [28,100), greedy takes the single best pair and strands the other two spans, finding one match where two exist. Under half-credit scoring that changes the reported F1.

What this tool does not do

It does not compute Krippendorff's unitizing alpha, which is the principled chance-corrected measure for this setting (Krippendorff, 1995) and considerably harder to interpret. It scores two annotators at a time, not a panel. Its tokeniser splits on whitespace and nothing else, which is deliberate — a deterministic rule anyone can reproduce is worth more in a diagnostic than correct punctuation handling. And the passage it loads is synthetic, written for this page: no real transcript content appears anywhere in these materials.

Frequently asked questions