How Many Annotators Per Item? Redundancy and the Cost Curve

How many annotators per item is the question every labeling budget eventually runs into, and the honest answer is that it depends on how noisy your labelers are—and it is almost never one. A single label gives you a number with no error bars. You cannot tell a careful judgment from a coin flip, and whatever noise is in that one label flows straight into your model or your gold set. The durable result from crowdsourcing research: once individual labels are noisy, repeated labeling with aggregation beats single labeling, even when labels are not free (Sheng, Provost & Ipeirotis, 2008).

More labels are not free, though, and they do not help forever. This post works through the three things that decide the number: why one label is risky, how aggregation methods—majority vote, Dawid-Skene, and MACE—turn many noisy labels into one better one, and where the accuracy-versus-cost curve flattens so you stop paying for labels that no longer move the answer.

How many annotators per item do you actually need?

There is no single number; the right count is a function of how accurate each labeler is and how much a wrong label costs you. If your labelers are near-perfect, one label is close to optimal and redundancy is waste. If they are noisy but better than chance, redundancy with aggregation reliably raises quality, and the noisier they are, the more labels you need to reach the same target (Sheng, Provost & Ipeirotis, 2008).

For a usable starting point, borrow a number from the literature. Studying five natural-language tasks on Mechanical Turk, Snow and colleagues found that an average of 4 non-expert labels per item was enough to emulate the quality of a single expert annotator on an affect-recognition task (Snow, O’Connor, Jurafsky & Ng, 2008). That is a specific, checkable figure for a specific, noisy pool—not a universal law, but a good order-of-magnitude prior.

The practical procedure is to stop guessing and measure. Collect 3 labels per item on a pilot batch, compute inter-rater agreement, and let the disagreement tell you where you stand—see how to compute Cohen’s kappa for the mechanics. If agreement is high, drop to fewer labels; if it is low, the fix is usually better guidelines before more labels.

Why one label per item is riskier than it looks

One label per item is risky because it is unfalsifiable: with a single judgment you have no way to know whether it is right, and no way to estimate your own error rate. Redundancy is what turns labels from assertions into measurements—two or more labels give you disagreement, and disagreement is the only cheap signal you have about which items are hard and which labelers are drifting.

That signal is worth more than it looks. Annotator disagreement is signal, not noise: the items your labelers split on are exactly the ambiguous cases your guidelines failed to define, and you cannot see them at all from a single label. Redundant labeling also surfaces the mislabeled examples that quietly cap model performance, which is why finding label errors in your dataset usually starts with items that had more than one labeler.

The cost of a single noisy label is not confined to that row. If it lands in your training set, it teaches the model the wrong boundary; if it lands in your evaluation set, it corrupts the number you use to judge every model after it. One label per item is cheapest per row and most expensive per mistake.

Does majority vote improve label quality?

Majority vote improves label quality only when each labeler is better than chance, and even then with sharply diminishing returns. If individual accuracy is above 50%, the majority of independent labelers grows more accurate as you add labels—the Condorcet jury theorem in practice. If accuracy is below 50%, the same math runs in reverse and more labels make the aggregate worse.

The shape of the gain is the part most teams underestimate. The curve is concave: the jump from 1 to 3 labels is the largest, and each pair of labels after that buys less. For a labeler who is right 70% of the time, a majority vote of 3 lifts accuracy to about 78%, a vote of 5 to about 84%, and a vote of 7 to about 87%—real gains, but each step smaller than the last.

Labels per item (majority vote)	Weak labeler (55% each)	Typical labeler (70% each)	Strong labeler (90% each)
1 (single label)	55%	70%	90%
3	57%	78%	97%
5	59%	84%	99%
7	61%	87%	99.7%
9	62%	90%	99.9%

Read the columns, not just the rows. A strong labeler is essentially done at 3 labels, so a fourth is wasted money. A weak labeler needs a pile of labels to gain a few points, which is often a sign you should fix the labeler—through screening and training or a clearer schema—rather than buy your way out with redundancy. Majority vote is transparent and hard to get wrong, but it assumes every annotator is equally competent, which is exactly where it starts to leak.

Majority vote vs Dawid-Skene vs MACE: which aggregation?

The three common aggregators differ in what they assume about your annotators. Majority vote assumes everyone is equally good; Dawid-Skene and MACE do not, which is why they can recover the truth in cases where majority vote cannot. When a few skilled annotators are outnumbered by a larger set of mediocre ones, counting votes hands the answer to the majority; modeling competence hands it to the people who are usually right.

Dawid-Skene, the older idea, estimates a full confusion matrix for each annotator—how often they call a true A a B, and so on—then uses the EM algorithm to infer the most likely true label given everyone’s patterns (Dawid & Skene, 1979). MACE takes a Bayesian route and adds an explicit model of spamming behavior, so it is unusually good at spotting annotators who are guessing or gaming the task and down-weighting them, while returning a competence score per worker (Hovy, Berg-Kirkpatrick, Vaswani & Hovy, 2013).

Aggregation method	What it models	Best when	Where it breaks
Single label (no aggregation)	Nothing—trusts one annotator	Labelers are near-perfect and budget is very tight	No way to catch or estimate error; noise flows straight into your data
Majority vote	All annotators equally competent; independent errors	Labelers are similar and better than chance; you want transparency	Spammers or a shared bias can outvote experts; ties on even counts
Dawid-Skene (EM)	Each annotator's full confusion matrix; infers the latent true label	Annotators vary in skill and some good ones are outnumbered	Needs enough labels per annotator; can settle on a local optimum (Dawid & Skene, 1979)
MACE (Bayesian)	Annotator trustworthiness plus a random-spamming strategy	You suspect spammers or adversarial workers and want competence scores	Coordinated colluders can still fool it; assumes independent errors (Hovy et al., 2013)

Does the extra machinery pay off? In a word-sense annotation case study, a probabilistic annotation model produced better labels than the conventional majority-style approach and did it at less than half the cost, while also returning a per-label certainty measure that majority vote cannot give you (Passonneau & Carpenter, 2014). The catch is that these models need enough labels per annotator to estimate their parameters and assume errors are independent given the true label—when annotators share a systematic bias, no aggregator fully rescues you.

The accuracy-vs-cost curve: where more annotators stop paying off

More annotators per item stop paying off at the point where the next label’s accuracy gain is worth less than the next label’s cost. Accuracy from redundancy is concave; cost is linear. So the value you actually capture rises, peaks, and then falls as you keep buying labels past the useful range.

Two failure modes bracket the sweet spot. Over-labeling strong annotators buys near-zero accuracy for full price: if your pool is right 90% of the time, three labels already reach ~97% and a fifth barely moves the number. Under-labeling weak annotators leaves quality on the table, but the honest fix is rarely “keep adding labels”—a pool that needs nine labels to reach 62% is telling you the task or the training is broken.

The most useful move in the original repeated-labeling work is to stop labeling everything the same number of times. Selective repeated labeling—concentrating extra labels on the items where labelers currently disagree, and leaving the confident items alone—achieved higher quality for the same budget than uniform round-robin relabeling (Sheng, Provost & Ipeirotis, 2008). You get the redundancy where it changes the answer and skip it where it would not.

How many annotators per item for subjective clinical coding?

Subjective clinical coding is the regime where redundancy matters most, because individual accuracy is well below 100% and the constructs are genuinely hard. Rating depressive severity from an interview on the ten-item, clinician-rated MADRS (Montgomery & Åsberg, 1979) or scoring formal thought disorder on the TLC (Andreasen, 1986) is exactly the “noisy but better than chance” case where multiple raters plus aggregation earns its keep—and exactly where a single rater’s judgment should not become ground truth unchecked.

There is a caveat that changes the arithmetic. Trained clinical raters often share the same bias—they were taught the same anchors—so their errors are correlated, and correlated errors are the one thing majority vote and even model-based aggregation handle poorly. On these tasks, the highest-return spend is upstream: anchored definitions, calibration, and training, which is why pay alone rarely improves annotation quality and why the expert-versus-crowd tradeoff looks different for clinical scales than for generic labeling. Redundancy is a multiplier on labelers who already share a good definition, not a substitute for one.

Where Tagaroo fits

Tagaroo is built for the workflow this evidence points to: define a coding scheme once with anchored definitions and examples, let an AI agent take a first pass, and have human reviewers confirm or correct it with redundant labeling, inter-rater reliability, and adjudication in the loop. That puts the budget where the curve says quality lives—on the contested items and on the guidelines—rather than on a flat “N labels for everything” rule.

Its boundary is worth stating too. If your goal is to push millions of easy microtasks through the cheapest possible pool, a bare crowdsourcing marketplace with plain majority vote is a different, lighter tool. Tagaroo’s strength is the guided middle: subjective, schema-driven coding where aggregation and reliability are measured, not assumed. On data handling it takes a de-identify-first path—strip direct identifiers before upload, with an anonymous browser-side trial mode so trial data never leaves your machine; see the privacy policy for specifics.

The practical upshot

The question “how many annotators per item” has no fixed answer, but it has a fixed method: estimate how accurate your labelers are, start at 3 labels, measure agreement, and then spend redundancy selectively on the items that split. Aggregate with majority vote when your pool is uniform and honest, and reach for Dawid-Skene or MACE when skill varies or spammers lurk (Dawid & Skene, 1979; Hovy et al., 2013). Above all, watch the curve: for strong labelers it flattens by the third label, and past the flattening point you are buying confidence you already had.

If you change one thing, change this: stop labeling every item the same number of times. Concentrate your labels where the disagreement is, fix the guidelines when redundancy stops helping, and define your scheme and calibrate your raters in Tagaroo so redundancy multiplies a shared definition instead of papering over the lack of one. For the imaging version of the same question, see how many annotators you need for medical images.

References

• Sheng VS, Provost F, Ipeirotis PG. Get Another Label? Improving Data Quality and Data Mining Using Multiple, Noisy Labelers. Proceedings of the 14th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD ’08). 2008:614–622. doi:10.1145/1401890.1401965
• Dawid AP, Skene AM. Maximum Likelihood Estimation of Observer Error-Rates Using the EM Algorithm. Journal of the Royal Statistical Society: Series C (Applied Statistics). 1979;28(1):20–28. doi:10.2307/2346806
• Hovy D, Berg-Kirkpatrick T, Vaswani A, Hovy E. Learning Whom to Trust with MACE. Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT). 2013:1120–1130. aclanthology.org/N13-1132
• Snow R, O’Connor B, Jurafsky D, Ng AY. Cheap and Fast—But is it Good? Evaluating Non-Expert Annotations for Natural Language Tasks. Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing (EMNLP). 2008:254–263. doi:10.3115/1613715.1613751
• Passonneau RJ, Carpenter B. The Benefits of a Model of Annotation. Transactions of the Association for Computational Linguistics. 2014;2:311–326. doi:10.1162/tacl_a_00185