How Many Annotators Medical Images Need: A Consensus Guide

How many annotators medical images need is not a single number, and any vendor who quotes you one is selling a workflow, not an answer. The honest answer is that the read count is a function of clinical risk and of what your label has to be true against. A quality-control flag on a triage model and a tumor-margin mask that feeds a treatment measurement should not get the same number of eyes, and spending equal effort on both wastes your experts on the easy half.

Here’s the shape of it, up front. One read is an opinion; ground truth for anything consequential comes from combining several independent reads and then adjudicating where they disagree (Willemink et al., 2020). This post is the deep-dive companion to our medical image annotation guide: it works through single versus multi-reader designs, blind parallel reads, adjudication, consensus scoring, and how to tie your review rate to risk with a table you can lift into a project plan.

How many annotators medical images need: the short answer

The number of annotators a medical image needs rises with clinical risk: one spot-checked read for low-risk, well-specified labels; two blind reads with a tie-breaker for medium-risk detection; three or more blind expert reads with mandatory adjudication for boundary-sensitive findings that inform treatment. The reference standard sets the floor, because expert reads only count as ground truth when imaging itself is definitive (Willemink et al., 2020).

The table below is the whole recommendation in one place. It’s deliberately built to be quotable and to drop straight into a data-management plan.

Clinical risk	Example task	Reads per image	Consensus rule	Adjudication	Human review rate
Low	Image-quality flags, coarse triage, normal-vs-abnormal pre-sort	1 read	None	None	Spot-check 5–10%
Medium	Lesion detection, counting, well-specified classes	2 blind reads, +1 on conflict	2-of-3 majority	Senior review on disagreement	20–30%
High	Tumor margins, volumes, measurements that inform treatment	≥3 blind expert reads	STAPLE / median / majority	Mandatory senior adjudication	100% of discordant cases + a concordant sample

Two design choices sit underneath the whole table. First, the reads have to be blind and independent, or the consensus is theater. Second, you need an escalation rule fixed before labeling starts, so an adjudicator engages only on genuine disagreement rather than re-reading everything. The rest of this post is why those two choices carry so much weight.

Why is one reader rarely enough?

One reader is rarely enough because a single read carries that reader’s blind spots and bad day with no way to catch them, and for consequential findings the miss rate is not academic. The cleanest evidence comes from breast screening, where double reading is standard of care in several national programs and has been measured at scale.

In a prospective study of 99,013 digital screening mammograms read in a blinded double-reading design, adding a second reader raised the cancer detection rate from 6.2 to 7.0 per 1,000 screens, a 13.3% relative increase, and caught 82 additional cancers the first reader missed (Coolen et al., 2018). That gain isn’t free: the recall rate rose from 3.0% to 3.6% and the false-positive rate from 24.4 to 28.7 per 1,000 (Coolen et al., 2018). A second reader buys sensitivity and charges you specificity.

There’s a theory under the empirical result. Sheng, Provost, and Ipeirotis showed formally that when labels are noisy, repeated labeling can beat single labeling even when reads aren’t cheap, because independent errors tend to cancel under a vote (Sheng, Provost & Ipeirotis, 2008). The catch, which the next section is about, is the word independent.

What are blind parallel reads, and why blind?

Blind parallel reads mean several annotators label the same image at the same time, each without seeing the others’ work, and only afterward do you compare and combine. Blinding is not a bureaucratic nicety. It’s the assumption that makes a consensus vote mean anything.

The math only works if errors are uncorrelated. If your second reader can see the first read, two things happen: easy anchoring inflates agreement, and the reads stop being independent samples of the truth. The mammography evidence above was specifically a blinded double-reading design for that reason (Coolen et al., 2018), and crowd-labeling work makes the same assumption explicit when it treats repeated labels as independent draws (Sheng, Provost & Ipeirotis, 2008).

Here’s where it gets subtle. Perfect independence is a fiction: readers share training, share priors, and get fooled the same way by the same genuinely ambiguous image. When errors correlate, a majority vote helps far less than the clean theory promises, and no amount of extra readers rescues a case that is inherently unclear.

That residual is not noise to scrub. It’s the irreducible disagreement we come back to under adjudication.

A synthetic worked example: what a third reader buys you

To make the independence assumption concrete, here’s a small synthetic calculation. Suppose one reader labels a binary finding (“nodule: yes/no”) with a 10% error rate, and you send each image to three independent readers and take the majority. This is illustrative arithmetic on synthetic numbers, not a dataset.

A 2-of-3 majority is wrong only when at least two of the three reads are wrong. With independent errors at 10%, that probability is 3 × (0.10)² × (0.90) + (0.10)³ = 2.8%. The error rate falls from 10% to under 3% by adding two reads, which is the repeated-labeling benefit formalized by Sheng, Provost, and Ipeirotis (2008).

Now break the assumption, which is the realistic case. If the three readers share a systematic blind spot, so their errors are correlated rather than independent, the majority error climbs back toward the single-reader rate, because they tend to be wrong together on the same hard images. That’s the quantitative reason blinding matters and the reason a stubborn minority of cases survive any number of reads. It also tells you where your effort should go: not into a fourth routine read, but into adjudicating the cases that split.

How do you turn multiple reads into one label?

You combine them with a rule chosen before labeling, matched to the annotation type, because majority voting is meaningless for some outputs. Pre-specifying the rule is the point; arguing it after you see the disagreements invites bias (Innolitics, 2025).

Three patterns cover most medical-imaging projects:

1. Categorical labels (finding present or absent, a class): a majority vote, commonly 2-of-3 across three blinded readers, with an adjudicator engaged only on a split (Encord, 2022; Innolitics, 2025).
2. Measurements (a diameter, a volume): the median of the reads, or the two most concordant values, escalating only when readers exceed a pre-set gap.
3. Segmentation masks: don’t vote on boundaries by hand. Run STAPLE, which estimates a probabilistic consensus segmentation and each rater’s performance level simultaneously (Warfield et al., 2004), or fall back to per-pixel majority.

The same batch of images can need different rules as the task changes. Three readers screening for a nodule use a 2-of-3 vote; the moment the task becomes segmenting that nodule’s volume, majority voting on a pixel boundary is nonsense and you feed the masks to STAPLE instead. How much agreement is “enough,” and how to measure spatial overlap, is its own subject. See image segmentation agreement with Dice, IoU, and STAPLE for masks and Cohen’s kappa for inter-rater reliability for categorical agreement.

How do you adjudicate disagreement, and what can’t be fixed?

Adjudication is the step where a senior reader resolves the cases your consensus rule flagged as discordant, and it does real work: structured adjudication has been shown to significantly revise treatment-relevant clinical parameters, not just tidy up edge cases (Schaekermann et al., 2019). The operative word is structured, meaning readers cite the specific guideline or definition behind their call rather than argue from vibes.

But the standard advice breaks down when you assume adjudication can converge everything. In a case study of 36 experts and 4,543 adjudicated cases in a sleep-staging task, Schaekermann and colleagues found that disagreement separates into two kinds: resolvable, where deliberation and a clearer guideline bring readers together, and irreducible, where genuine ambiguity in the data keeps experts apart no matter how much they talk (Schaekermann et al., 2019). Their earlier work drew the same line between resolvable and irresolvable disagreement in crowd deliberation (Schaekermann et al., 2018).

That distinction changes how you should read your disagreement rate. Recurring, resolvable disagreement usually means your codebook is ambiguous, not that a rater is careless, so the fix is a better definition, not a stricter reviewer. Irreducible disagreement is information about task difficulty and belongs in your uncertainty estimate; forcing it to a single answer manufactures false precision. Where readers are genuinely split after adjudication, an explicit “uncertain” label beats a coin-flip.

How much of the data should a human review?

Set the human review rate by clinical risk, not by a flat percentage across the dataset. A screening-triage flag and a segmentation mask that informs a treatment dose do not deserve equal scrutiny, and a uniform review rate over-audits the safe cases and under-audits the dangerous ones. Tie review intensity to what a wrong label would cost downstream, exactly as the risk table lays out.

In practice that means near-total review of high-risk discordant cases plus a sampled audit of the concordant ones, and a light spot-check on low-risk labels where a mistake is cheap and recoverable. Two operational habits pay off repeatedly regardless of tier: a pilot phase on a handful of cases to surface protocol ambiguity before it scales, and treating inter-annotator disagreement as a metric you watch, since a rising split rate is an early warning that your definitions have drifted. Model-assisted pre-labeling can shift where the human effort goes, letting reviewers correct rather than draw, which is its own topic in model-assisted image labeling.

Where curated coding schemes fit

The cheapest way to need fewer annotators is to remove the disagreement you can before anyone reads: a precise, anchored coding scheme converts a chunk of resolvable disagreement into simple agreement. If two readers apply the same operational definition the same way, you didn’t need a third read to break a tie that never happened.

Clinical measurement solved this decades ago, and imaging teams can borrow the pattern. A validated rating scale is a curated coding scheme with anchored definitions, a fixed structure, and a citation trail. Look at how the MADRS depression scale specifies each item with severity anchors, or how the TLC scale for thought, language and communication gives every item an operational definition with verbatim examples. That’s the bar an image-annotation data dictionary should aim for: every label defined, every foreseeable ambiguity pre-resolved.

When you must handle PHI-bearing source data on the way in, the de-identification mechanics live in preparing DICOM data for annotation, and a concrete end-to-end read-through is in our chest X-ray annotation example.

Where Tagaroo fits

Tagaroo is a schema-first annotation workspace: you define a coding scheme once, then tag against it with inter-rater reliability and adjudication built into the workflow rather than bolted on afterward. Blind parallel reads, a pre-specified consensus rule, and agreement computed as annotators work are the parts general-purpose tools tend to leave to a spreadsheet, and they’re exactly the parts that decide whether your multi-reader design produces defensible ground truth.

Be clear about the boundary. If your job is industrial-scale pixel segmentation of DICOM volumes with specialized auto-annotation, a dedicated computer-vision platform is the right call. Tagaroo’s strength is guided, schema-driven annotation with every label anchored to the evidence that justifies it, and the multi-rater mechanics this post argues for treated as first-class.

On data handling, Tagaroo takes the de-identify-first path this guide argues for, rather than positioning itself as a PHI processor: its terms require you to strip direct identifiers, DICOM header fields and burned-in pixel PHI, before upload; its anonymous trial mode runs entirely in the browser so trial images never leave your machine; and the stack is EU-hosted and GDPR-oriented. It is not a medical device, and any AI pre-label is a first pass for a qualified human to review, never a diagnosis. If your project must process raw PHI in the cloud under a signed Business Associate Agreement, put that question to any vendor, including this one, before uploading identifiable studies; see Tagaroo’s privacy policy for specifics.

The practical upshot

How many annotators medical images need is a risk decision, not a constant. Spot-check the low-risk labels, read the medium-risk findings twice with a tie-breaker, and give boundary-sensitive, treatment-informing work three blind expert reads with mandatory adjudication. Keep the reads blind so the consensus means something, fix the combination rule before labeling, and accept that a stubborn minority of disagreements are irreducible and belong in your uncertainty budget rather than your error log.

If you change one thing, change this: stop treating a single read as ground truth. Start from the coding scheme that makes agreement possible, and build your annotation scheme in Tagaroo with reliability and adjudication in the loop from the first label.

References

• Coolen AMP, Voogd AC, Strobbe LJA, et al. Impact of the second reader on screening outcome at blinded double reading of digital screening mammograms. British Journal of Cancer. 2018;119(4):503–507. doi:10.1038/s41416-018-0195-6
• Schaekermann M, Beaton G, Habib M, Lim A, Larson K, Law E. Understanding Expert Disagreement in Medical Data Analysis through Structured Adjudication. Proceedings of the ACM on Human-Computer Interaction. 2019;3(CSCW):Article 76. doi:10.1145/3359178
• Schaekermann M, Goh J, Larson K, Law E. Resolvable vs. Irresolvable Disagreement: A Study on Worker Deliberation in Crowd Work. Proceedings of the ACM on Human-Computer Interaction. 2018;2(CSCW):Article 154. doi:10.1145/3274423
• Sheng VS, Provost F, Ipeirotis PG. Get Another Label? Improving Data Quality and Data Mining Using Multiple, Noisy Labelers. Proceedings of the 14th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD ’08). 2008:614–622. doi:10.1145/1401890.1401965
• Warfield SK, Zou KH, Wells WM. Simultaneous Truth and Performance Level Estimation (STAPLE): An Algorithm for the Validation of Image Segmentation. IEEE Transactions on Medical Imaging. 2004;23(7):903–921. doi:10.1109/TMI.2004.828354
• Willemink MJ, Koszek WA, Hardell C, et al. Preparing Medical Imaging Data for Machine Learning. Radiology. 2020;295(1):4–15. doi:10.1148/radiol.2020192224
• Innolitics. Definitive Guide to AI/ML SaMD Ground Truthing: Adjudication Strategies. 2025. innolitics.com
• Encord. How to Structure QA Workflows for Medical Images. 2022. encord.com