Annotation Consensus Methods: Majority Vote to Dawid-Skene

Ten coders label the same interview turn. Six call it disorganized speech, four call it ordinary hesitation. Do you ship the majority and move on? That single decision, how you collapse many labels into one, is what annotation consensus methods are for, and picking the wrong one quietly bakes bias into the ground truth every downstream model learns from. Most teams reach for a majority vote by reflex. Sometimes that is exactly right. Often it throws away the very information the extra annotators were collected to provide.

What are annotation consensus methods?

Annotation consensus methods are the procedures that reconcile several annotators’ labels for the same item into a single reference label (a “gold” label), plus, in the better methods, an estimate of how much to trust each annotator and each resulting label. They sit at the end of the labeling pipeline: you collected multiple labels per item precisely so you could measure and correct for disagreement, and this is where that investment pays off or leaks away.

There are five families worth knowing, and they trade off cost, assumptions, and how much they tell you. Simple vote counting is cheap and assumes competent, independent raters. Weighted voting adds a reliability weight. Probabilistic models like Dawid-Skene and MACE infer both the true labels and each annotator’s error profile at once. STAPLE does the same job for spatial segmentation. Expert adjudication puts a human back in the loop for the items the others cannot settle.

The choice is not academic. Passonneau and Carpenter (2014) put it bluntly: standard agreement measures “are neither necessary nor sufficient to ensure a high quality corpus.” A high inter-rater reliability score does not tell you which label is right on a contested item, and a low one does not mean the data is unusable. The aggregation method is where you decide what “right” means.

Method	Use it when	What it assumes	Main failure mode
Majority / plurality vote	Errors are independent and roughly unbiased; labels are categorical; you want something fast	All annotators are equally competent	Ties; a systematically biased majority; a lone expert who is right
Weighted vote	You have a trustworthy weight (gold-item accuracy or known expertise)	The weights are current and the gold set is representative	Stale or gamed weights; unrepresentative gold items
Dawid-Skene (probabilistic EM)	Several labels per item across many items; you want consensus plus per-annotator error rates	Each annotator has a fixed confusion matrix; errors are conditionally independent given the true label	Too few labels per item; correlated errors shared across raters
MACE (competence estimation)	Crowd data where some raters may be adversarial or lazy	Raters are either answering faithfully or spamming a fixed strategy	Everyone wrong the same way; genuine ambiguity read as incompetence
STAPLE (segmentation)	Pixel or voxel masks, not category labels	Per-rater sensitivity/specificity; conditional independence per voxel	Correlated boundary errors; very few annotators
Expert adjudication / deliberation	High-stakes items, ties, or genuine ambiguity	A senior rater or panel can and should resolve it	Cost and non-scalability; some disagreement is irresolvable

When is majority vote enough, and when does it fail?

Majority vote is enough when annotator errors are independent and roughly unbiased, the labels are categorical, and no single rater carries decisive expertise the others lack. Under those conditions the crowd genuinely is wise, and the evidence is strong. Snow et al. (2008) found that averaging the labels of about four non-expert annotators matched the quality of a single expert across a range of natural-language tasks, and on a word-sense task, simple majority voting with random tie-breaking plateaued at 0.994 accuracy.

But majority vote makes an assumption it cannot check: that every voter is equally worth counting. Break that assumption and the method breaks with it, in three predictable ways.

First, ties. With an even number of annotators or a genuinely split item, majority vote has no answer and falls back to a coin flip, which is exactly where you least want randomness. Second, systematic bias. If most of your raters share the same misreading (a common outcome of the same weak guidelines or the same rushed training), the majority is confidently wrong and no amount of extra voters fixes it, because the errors are correlated rather than independent. Third, the lone-expert-is-right case: a subtle positive that only your two most careful raters catch will be outvoted by three who miss it.

Sheng, Provost and Ipeirotis (2008) made the boundary precise. Repeated labeling improves both label quality and model quality, they showed, but “not always”; the benefit depends on labeler quality and the cost regime, and selectively re-labeling the items you are most uncertain about beats labeling everything more times. In other words, majority vote is a strategy, not a default. Deciding how many labels to gather before you aggregate is its own question, covered in how many annotators you need per item.

What do weighted and probabilistic models add? (Dawid-Skene and MACE)

Probabilistic aggregation models add the thing majority vote lacks: a learned estimate of how much each annotator should count, inferred from the data instead of assumed. The foundational one is Dawid-Skene, proposed in 1979 to reconcile patient histories recorded by different clinicians, and still the workhorse of modern label aggregation (Dawid & Skene, 1979).

Dawid-Skene models each annotator with a confusion matrix: the probability that they report label B when the true label is A, for every pair. It then runs expectation-maximization, alternating between two estimates it does not know in advance. In the E-step, it estimates the probability of each item’s true label given the current guesses about annotator error rates. In the M-step, it re-estimates every annotator’s confusion matrix given the current probabilistic labels. The loop converges on a soft consensus label per item plus a reliability profile per annotator. A rater who systematically confuses two categories is down-weighted for exactly those categories, not globally.

That per-annotator, per-category weighting is why the model beats a show of hands. Passonneau and Carpenter (2014) applied a Dawid-Skene-style probabilistic model to crowdsourced word-sense data and reported two results worth keeping in mind: the model produced high-confidence labels even for items where chance-adjusted agreement among trained annotators was low, and it attached a certainty measure to every gold label, all from data collected at less than half the cost of the conventional trained-annotator pipeline.

MACE (Multi-Annotator Competence Estimation) tackles the same problem from a slightly different angle. Hovy et al. (2013) framed it as an item-response model that learns, unsupervised, which annotators are trustworthy and what the correct labels are, and it was built specifically for the crowdsourcing failure mode where “some annotators choose bad labels in order to maximize their pay.” MACE is the tool to reach for when you suspect adversarial or lazy raters, because it explicitly models a spammer as someone following a fixed answering strategy rather than reading the item.

How do you build consensus for image segmentation? (STAPLE)

Category-based voting does not work for image segmentation, because a pixel mask is a spatial object, not a single categorical choice. The standard consensus method there is STAPLE (Simultaneous Truth and Performance Level Estimation), which is essentially Dawid-Skene’s logic applied to voxels. It takes several segmentations of the same structure and, by expectation-maximization, estimates both a probabilistic consensus mask and each annotator’s sensitivity and specificity (Warfield, Zou & Wells, 2004).

The motivation is the same one that drives probabilistic label aggregation. As Warfield and colleagues put it, “vote counting strategies treat each voter equally without regard to potential variability in quality or performance amongst the voters.” STAPLE instead learns who to trust from the overlap pattern and weights each mask accordingly, so a rater who systematically under-traces an edge is discounted rather than allowed to erode the consensus contour. The full treatment, including why Dice and IoU replace kappa on pixels, is in the guide to image segmentation agreement with Dice, IoU, and STAPLE.

When should you use expert adjudication instead?

Use expert adjudication when disagreement reflects genuine difficulty rather than annotator error, when the item is high-stakes enough that a wrong gold label is costly, or when an automatic method flags an item as low-confidence and cannot resolve it. Adjudication means a senior rater, or a small panel, reviews the contested item, sees the competing labels and their rationales, and issues a decision, often after discussion.

The key insight is that not all disagreement is the same. Schaekermann et al. (2018) draw a sharp line between resolvable and irresolvable disagreement. Resolvable disagreement comes from a misread, a missed guideline clause, or a slip, and their study found that structured deliberation among workers can settle many such cases and improve accuracy over pure aggregation. Irresolvable disagreement is different: it arises “for legitimate reasons and thus carries valuable information,” and forcing it to a single label destroys signal you might want. The practical move is to record those items as genuinely ambiguous rather than pretend a majority settled them. That disagreement is often a feature, not a defect, is the argument in why disagreement is signal, not noise.

Adjudication is accurate and expensive, so its place in a real workflow is targeted, not universal. Let a cheap method (majority vote or Dawid-Skene) label everything, have that method surface the items where confidence is low or a spammer was outvoted, and route only those to an expert. That two-tier design is why expert judgment and crowd scale are complements rather than rivals, a tradeoff explored in expert vs crowd annotation.

A worked example: five coders, one hard item

Consider a synthetic case that shows why the method matters. Five coders rate whether a single interview turn shows perseveration (yes or no). The item is a subtle true “yes.” Two experienced clinical coders catch it; three crowd raters, who have consistently missed subtle positives across the batch, vote “no.”

Coder	This item's vote	Track record (from prior items)	How each method treats the vote
C1 (experienced)	Yes	High sensitivity on subtle cases	Majority: 1 vote · Dawid-Skene: high weight
C2 (experienced)	Yes	High sensitivity on subtle cases	Majority: 1 vote · Dawid-Skene: high weight
R1 (crowd)	No	Often misses subtle 'yes'	Majority: 1 vote · Dawid-Skene: 'no' down-weighted
R2 (crowd)	No	Often misses subtle 'yes'	Majority: 1 vote · Dawid-Skene: 'no' down-weighted
R3 (crowd)	No	Often misses subtle 'yes'	Majority: 1 vote · Dawid-Skene: 'no' down-weighted

Straight majority vote returns three “no” against two “yes” and confidently labels the item negative. Dawid-Skene sees the same five votes but weights them by each coder’s estimated confusion matrix: because R1 through R3 have a documented habit of missing subtle positives, their “no” carries little evidence about a subtle case, and the model’s posterior tips to “yes.” Same votes, opposite gold label.

Two honest caveats keep this from being magic. Dawid-Skene needs enough items overall to estimate those confusion matrices; on a single isolated item it has nothing to learn from. And if all three crowd raters miss the case for the same reason your guideline is unclear, that is correlated error, and no aggregation method will save you. When aggregation and adjudication both flag the same items, that overlap is often where real label errors hide, which is the starting point for finding label errors in your dataset.

How does this map to clinical transcript coding?

The same split shows up the moment two clinicians code the same transcript. Deciding what label a passage gets, rating overall depression severity against the MADRS, say, is a categorical judgment where majority vote or a Dawid-Skene consensus is the honest way to combine raters. Deciding where a phenomenon occurs, marking the exact span of disorganized speech against the Thought, Language, and Communication scale, is a boundary-overlap problem closer to the segmentation case, where a spatial consensus makes more sense than a category vote.

High-stakes clinical items are also where the resolvable-versus-irresolvable distinction earns its keep. A borderline severity rating that two raters split on may be genuinely ambiguous, and a psychiatric research team is usually better served recording that ambiguity, and routing it to a senior adjudicator, than letting a coin-flip tie-break decide it. Any of this only works on de-identified or synthetic transcript text; consensus math is no substitute for handling the underlying data responsibly.

Tagaroo is built for the categorical-and-span side of that world: it collects multiple coders’ labels on the same transcript, computes inter-rater reliability as they work, and gives you the raw per-annotator labels you need to run any of the aggregation methods above, or to route a contested item to an expert. The tool does not decide the method for you. That is your call, and it should follow the task.

Which annotation consensus method should you use?

Match the method to what the labels are and how much a wrong gold label costs. For independent, unbiased categorical labels where speed matters, majority vote is fine, and roughly four labels per item is a sensible starting budget (Snow et al., 2008). When you have many items with several labels each and want to correct for unreliable raters, Dawid-Skene gives you a consensus plus a per-annotator error profile; MACE is the sharper tool when you suspect spammers (Dawid & Skene, 1979; Hovy et al., 2013). For segmentation masks, use STAPLE (Warfield et al., 2004). And for the genuinely hard or high-stakes items, spend the money on expert adjudication, while recording the disagreement you cannot resolve (Schaekermann et al., 2018).

If you change one thing, change this: stop treating majority vote as the default and start treating it as one choice among several, each with an assumption you can check. The annotation consensus methods that report a per-label certainty and a per-annotator reliability do not just hand you a gold label; they tell you which of those labels to believe. Pick the aggregation for the task, not the other way round, and route the uncertain remainder to a human before it becomes training data. When you write up the result, report the reliability and the resolution rule together so a reader knows not just how much your coders agreed, but how you decided when they did not.

References

• Dawid, A. P., & Skene, A. M. (1979). Maximum likelihood estimation of observer error-rates using the EM algorithm. Journal of the Royal Statistical Society: Series C (Applied Statistics), 28(1), 20–28. doi.org/10.2307/2346806
• Sheng, V. S., Provost, F., & Ipeirotis, P. G. (2008). Get another label? Improving data quality and data mining using multiple, noisy labelers. Proceedings of the 14th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 614–622. doi.org/10.1145/1401890.1401965
• Snow, R., O’Connor, B., Jurafsky, D., & Ng, A. Y. (2008). Cheap and Fast—But is it Good? Evaluating Non-Expert Annotations for Natural Language Tasks. Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing (EMNLP), 254–263. aclanthology.org/D08-1027
• Hovy, D., Berg-Kirkpatrick, T., Vaswani, A., & Hovy, E. (2013). Learning Whom to Trust with MACE. Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), 1120–1130. aclanthology.org/N13-1132
• Passonneau, R. J., & Carpenter, B. (2014). The Benefits of a Model of Annotation. Transactions of the Association for Computational Linguistics, 2, 311–326. doi.org/10.1162/tacl_a_00185
• Warfield, S. K., Zou, K. H., & Wells, W. M. (2004). Simultaneous truth and performance level estimation (STAPLE): an algorithm for the validation of image segmentation. IEEE Transactions on Medical Imaging, 23(7), 903–921. doi.org/10.1109/TMI.2004.828354
• Schaekermann, M., Goh, J., Larson, K., & Law, E. (2018). Resolvable vs. Irresolvable Disagreement: A Study on Worker Deliberation in Crowd Work. Proceedings of the ACM on Human-Computer Interaction, 2(CSCW), Article 154. doi.org/10.1145/3274423
• Landis, J. R., & Koch, G. G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33(1), 159–174. doi.org/10.2307/2529310