Image Segmentation Agreement: Dice, IoU, and STAPLE

Two radiologists outline the same lesion and their masks overlap almost perfectly on the empty background around it. Score that with Cohen’s kappa or raw percent agreement and you get a number near 1.0, even when the two contours disagree by a fifth of the lesion. That is the trap at the center of image segmentation agreement: the metric everyone trusts for categorical labels, kappa, is the wrong instrument for pixels, because the background it rewards is not the thing you care about. Segmentation agreement is a spatial-overlap problem, and it wants spatial-overlap tools: Dice, IoU, and, for building consensus ground truth from many annotators, the STAPLE algorithm.

Why doesn’t Cohen’s kappa work for image segmentation?

Cohen’s kappa fails on segmentation because it scores every pixel as an independent category and then rewards agreement on the background, which typically fills more than 95% of a medical image. Kappa was built to correct raw agreement for chance using the Landis and Koch (1977) bands, but that chance correction is exactly what breaks under extreme class imbalance.

Start with the simpler failure. Treat a mask as a per-pixel “lesion vs background” classification and compute plain percent agreement: two annotators will agree on almost every background pixel by default, so the number floats near 100% whether or not their contours line up. Raw agreement is uninformative here, which is the whole reason chance-corrected statistics exist.

Kappa is supposed to fix that, and it partly does, but it introduces a subtler problem the segmentation literature keeps rediscovering: the prevalence paradox. Feinstein and Cicchetti (1990) showed that a high observed agreement can collapse to a low kappa when the marginal totals are imbalanced, and a small structure on a large background is imbalance in its purest form. The practical consequence is that kappa depends on something that has nothing to do with annotation quality: how much empty background you happen to include.

The team behind the most-cited segmentation IAA study puts the underlying difficulty plainly: “it is challenging to assess agreement when multiple experts draw boundaries around regions in images,” because “human annotators are more likely to generate different contours when they are asked to annotate the outline” (Yang et al., 2023). Boundary disagreement is not categorical disagreement, and a category statistic cannot see it.

What happens to kappa when you crop the image?

Kappa moves; the overlap metrics do not. This is the cleanest way to see why kappa is the wrong tool, and it is worth working through with numbers.

Take a synthetic case. Two annotators, A and B, outline the same lesion on one slice. A’s mask covers 100 pixels, B’s covers 120 pixels, and they overlap on 90 pixels. That fixes the overlap metrics immediately, because neither one looks at the background:

Now score the same two masks with percent agreement and kappa, and vary only the field of view. On a full 200×200 slice (40,000 pixels), the two annotators agree on 90 shared lesion pixels plus 39,870 shared background pixels, so raw agreement is 99.9% and kappa works out to about 0.82. Crop tightly to a 150-pixel window around the lesion and the exact same annotations give kappa 0.33. Crop to the union itself, with no background at all, and kappa turns negative at about −0.13.

Nothing about the annotators’ work changed between those numbers. Only the amount of background did. Kappa and percent agreement both count true negatives, so they inherit the background as a free variable; Dice and IoU exclude true negatives by construction, which is precisely why they are the honest measure of a contour.

Reporting Cohen’s kappa on a segmentation is not a rounding-error mistake. It is measuring the wrong thing.

How do you measure image segmentation agreement?

You measure image segmentation agreement with a spatial-overlap metric, almost always the Dice similarity coefficient or its close twin the Jaccard index (IoU). Both take the two masks A and B and reward their intersection, and both ignore the background entirely, so they answer the question kappa cannot: how much do these two contours actually coincide?

The Dice coefficient is twice the overlap over the total mask size, 2·|A∩B| / (|A|+|B|), and in classification terms it equals the F1 score, the harmonic mean of precision and recall, because both reduce to 2·TP / (2·TP + FP + FN) (Taha & Hanbury, 2015). IoU is the overlap over the union, |A∩B| / |A∪B|.

Dice is always at least as large as IoU for any partial overlap, and the two convert exactly by Dice = 2·IoU / (1+IoU). Because that mapping is monotonic, the two never disagree about which of two segmentations is better; the choice between them is a reporting convention, not a measurement decision. The companion guide on Dice vs IoU vs Hausdorff distance works through that identity and the size-sensitivity trap in detail.

For more than two annotators, the standard agreement summary is the mean pairwise overlap: average the Dice or IoU across all n(n−1)/2 annotator pairs. Three annotators give three pairs; five give ten. Yang et al. (2023) report exactly this alongside a consensus estimate, using IoU together with per-annotator sensitivity and specificity to characterize both how much the group agrees and where individuals drift.

Metric	Counts background (TN)?	Formula	Fit for segmentation agreement
Percent agreement	Yes	(TP+TN) / total pixels	Poor: inflates to ~99% on any image
Cohen's / Fleiss' kappa	Yes	(p₀ − pₑ) / (1 − pₑ)	Poor as headline; prevalence-dependent
Dice (DSC)	No	2·|A∩B| / (|A|+|B|)	Strong: the default overlap metric in medical imaging
IoU (Jaccard)	No	|A∩B| / |A∪B|	Strong: the CV benchmark standard; ranks like Dice
STAPLE	Modeled	EM over per-rater sensitivity/specificity	Strong for consensus ground truth + per-rater scoring

What is the STAPLE algorithm, and how does it build consensus?

STAPLE (Simultaneous Truth and Performance Level Estimation) takes a collection of segmentations of the same structure and, in one pass, estimates two things at once: a probabilistic consensus of the hidden true segmentation, and a performance level for each annotator (Warfield, Zou & Wells, 2004). It is the principled way to turn several disagreeing masks into one reference and a per-rater scorecard.

The mechanism is an expectation-maximization loop. In the E-step, STAPLE estimates the probability that each voxel is truly part of the structure, given the current guesses about how good each annotator is. In the M-step, it updates each annotator’s sensitivity (the true-positive fraction, how reliably they mark real lesion pixels) and specificity (the true-negative fraction, how reliably they leave background alone), given the current probabilistic truth. The two steps alternate until they converge on a soft consensus mask that can be thresholded into a binary reference.

The reason this beats a show-of-hands is that a majority vote is credulous. As Warfield and colleagues put it, “vote counting strategies treat each voter equally without regard to potential variability in quality or performance amongst the voters” (Warfield, Zou & Wells, 2004). STAPLE instead learns who to trust from the data, and weights each annotator’s vote by their estimated sensitivity and specificity.

Work a synthetic three-annotator example. Annotators A and B are experienced; annotator C is cautious and systematically under-marks the lesion edge, so C misses real pixels the other two catch.

Annotator	Est. sensitivity	Est. specificity	Effect on the STAPLE consensus
A (experienced)	0.94	0.999	High trust; votes carry near-full weight
B (experienced)	0.91	0.998	High trust; agrees with A on the edge
C (cautious)	0.68	0.9997	Low sensitivity down-weighted; missed edge pixels not read as truth

A straight majority-of-three would let C’s misses erode the consensus edge wherever only one experienced annotator caught a pixel. STAPLE, having estimated C’s low sensitivity, discounts those disagreements automatically. This is why medical-imaging groups use STAPLE to generate training ground truth: it produces both the fused label and a quantitative read on each annotator’s reliability (Yang et al., 2023).

How many annotators and metrics should you report?

Report overlap first, consensus second, and per-rater performance third. No single number covers a multi-annotator segmentation task honestly, and the guideline literature is explicit that reporting one metric in isolation is a known source of statistical bias (Müller, Soto-Rey & Kramer, 2022).

• Primary: mean pairwise Dice or IoU. This is your agreement headline. Name which one you used, because a Dice of 0.85 and an IoU of 0.85 are not the same result.
• Consensus: a STAPLE reference plus per-annotator sensitivity and specificity. The fused mask becomes your ground truth; the scorecard tells you which annotators to retrain or audit.
• Boundary check when contours matter. Overlap metrics stay high while an edge is visibly wrong, so on thin or tubular structures add a boundary metric such as HD95, as the Metrics Reloaded consortium recommends selecting metrics from the task rather than from habit (Maier-Hein et al., 2024).
• Kappa only as a guarded secondary. Fleiss’ kappa can accompany the overlap metrics for a per-finding categorical label, but it should never be the pixel-agreement headline (Yang et al., 2023).

There is a real-world reason to measure all of this carefully: inter-annotator agreement is a ceiling, not a footnote. As Yang et al. (2023) note, “variability between expert annotators introduces noise in training data that can adversely impact the performance of AI algorithms,” and “human inter-annotator agreement defines the upper limit on our ability to measure automated performance.” A model cannot be trusted past the agreement of the humans who taught it. Getting clean, de-identified inputs in front of those humans is a prerequisite, which is where a DICOM annotation and de-identification pipeline and the broader medical image annotation guide come in.

How does this map to transcript and scale annotation?

The overlap-versus-category split is not unique to images; it reappears the moment two people annotate the same transcript. Marking where something occurs in text is a span-overlap problem, the one-dimensional cousin of IoU: two coders who both flag a passage but disagree on its exact start and end have partial overlap, scored just like two masks. Deciding what label a passage gets is a categorical judgment, and that is where chance-corrected agreement belongs.

Both live in clinical coding. Tagging the exact stretch of disorganized speech against the Thought, Language, and Communication scale is a boundary-and-overlap question, where token overlap plays the role Dice and IoU play for pixels. Rating overall depression severity against the MADRS is a categorical judgment, where a chance-corrected statistic like Cohen’s kappa is the honest measure. If you are computing kappa on those categorical ratings, our guide to Cohen’s kappa for inter-rater reliability covers when weighted kappa or Krippendorff’s alpha is the better fit.

Tagaroo works the language side of that split: it turns a coding scheme into guided span annotation with inter-rater reliability computed as coders work, a different job from labeling pixels. The statistical logic is the same one this post argues for, though: pick the agreement measure from the shape of the annotation, not from habit.

Which image segmentation agreement method should you use?

Choose the method from what the annotation is. For pixel masks, that means overlap metrics as the default and a category statistic almost never.

• Report Dice or IoU as your primary image segmentation agreement metric. They rank identically, so use whichever your field expects and name it explicitly. IoU dominates computer-vision benchmarks; Dice dominates medical imaging.
• Summarize multiple annotators with mean pairwise overlap, and build ground truth with STAPLE so you also get per-annotator sensitivity and specificity.
• Add a boundary metric when the edge carries the clinical meaning, because a high Dice can hide a wrong contour (Maier-Hein et al., 2024).
• Do not headline Cohen’s kappa on pixels. The background makes it prevalence-dependent and unstable; save kappa for genuinely categorical labels.

The practical upshot: segmentation agreement is a spatial question, so measure it with spatial tools. Dice and IoU tell you how much two annotators overlap; STAPLE tells you what their consensus is and who to trust; and a boundary metric catches the errors overlap hides. Reach for kappa on a mask and you will mostly be measuring how much empty space you drew a box around.

References

• Warfield, S. K., Zou, K. H., & Wells, W. M. (2004). Simultaneous truth and performance level estimation (STAPLE): an algorithm for the validation of image segmentation. IEEE Transactions on Medical Imaging, 23(7), 903–921. doi.org/10.1109/TMI.2004.828354
• Yang, F., Zamzmi, G., Angara, S., Rajaraman, S., Aquilina, A., Xue, Z., Jaeger, S., Papagiannakis, E., & Antani, S. K. (2023). Assessing Inter-Annotator Agreement for Medical Image Segmentation. IEEE Access, 11, 21300–21312. doi.org/10.1109/ACCESS.2023.3249759
• Feinstein, A. R., & Cicchetti, D. V. (1990). High agreement but low kappa: I. The problems of two paradoxes. Journal of Clinical Epidemiology, 43(6), 543–549. doi.org/10.1016/0895-4356(90)90158-L
• Maier-Hein, L., Reinke, A., Godau, P., et al. (2024). Metrics reloaded: recommendations for image analysis validation. Nature Methods, 21, 195–212. doi.org/10.1038/s41592-023-02151-z
• Müller, D., Soto-Rey, I., & Kramer, F. (2022). Towards a guideline for evaluation metrics in medical image segmentation. BMC Research Notes, 15, 210. doi.org/10.1186/s13104-022-06096-y
• Taha, A. A., & Hanbury, A. (2015). Metrics for evaluating 3D medical image segmentation: analysis, selection, and tool. BMC Medical Imaging, 15, 29. doi.org/10.1186/s12880-015-0068-x
• Landis, J. R., & Koch, G. G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33(1), 159–174. doi.org/10.2307/2529310