How to Find Label Errors in Dataset Quality Audits

Roughly 3.3% of the labels in ten of the most-cited machine-learning benchmarks are wrong, rising to about 6% of ImageNet’s validation set (Northcutt, Athalye & Mueller, 2021). These are the datasets the field trusts most. If the canonical benchmarks are that noisy, your in-house dataset almost certainly is too—and the errors are not randomly harmless. They can quietly flip which model you decide is best.

This guide is about how to find label errors in dataset audits before they cost you a decision. It covers why label errors are so common, the three signals that flag a suspect label, the confident-learning method behind the cleanlab library, and the human adjudication step that turns a flag into a fix.

Why are label errors so common in real datasets?

Label errors are common because labeling is a human process under time pressure, and every source of pressure leaves a residue of wrong labels. Frénay and Verleysen’s survey of label noise catalogs the usual causes: insufficient information given to the labeler, genuine expert mistakes, ambiguous or subjective categories, and low-quality or automated labeling at scale (Frénay & Verleysen, 2014). None of these require carelessness. Careful people mislabel ambiguous items every day.

The uncomfortable part is that even gold-standard benchmarks carry this noise. The ImageNet validation set—used for a decade to rank image models—contains an estimated 2,900 label errors, about 6% of it (Northcutt, Athalye & Mueller, 2021). If a dataset that thousands of papers depend on is 6% wrong, a dataset your team labeled in six weeks under deadline is not going to be cleaner by accident.

It helps to separate two things people lump together. A label error is a data point that is wrong under the scheme’s own rules. Ambiguity is a data point where two qualified coders can reasonably land differently because the item genuinely sits on a boundary. Both produce disagreement, but only one is fixable by correction—which is why the detection step and the adjudication step are different jobs.

What are the three signals of a label error?

Three signals surface most label errors, and the strongest audits use all three because each catches what the others miss. None is proof on its own; each is a reason to look.

1. Annotator disagreement. When two independent coders assign different labels to the same item, at least one is wrong or the item is ambiguous. Disagreement is the oldest and cheapest error detector, and it is the basis of ensemble-filtering methods that use multiple classifiers to flag suspect labels (Brodley & Friedl, 1999).
2. Model-confidence mismatch. When a reasonably calibrated model confidently predicts a different label than the one recorded, the recorded label is suspect. This is the core intuition behind confident learning (Northcutt, Jiang & Chuang, 2021).
3. Near-duplicate conflict. When two near-identical items carry different labels, at least one is wrong by definition. Duplicates and near-duplicates are a mechanical, high-precision way to catch inconsistency that neither humans nor models flag on their own.

Signal	What it catches	How to act on it	Failure mode
Annotator disagreement	Ambiguous items and outright mistakes	Route to a third coder or an adjudicator	Silent when both coders share the same misconception
Model-confidence mismatch	Confident, systematic mislabels	Rank by confidence, review the top of the list	Inherits the model's blind spots; miscalibration misleads
Near-duplicate conflict	Inconsistent labels on the same content	Reconcile the pair to one label	Only fires where duplicates exist

Notice the pattern in that last column. Disagreement goes blind when both coders share a wrong mental model. Model confidence goes blind exactly where the model is weak—the hard cases you most want to check. Near-duplicate conflict only fires where duplicates exist.

Run all three and the blind spots stop overlapping. That is the whole argument for a layered audit over a single clever trick.

How does confident learning find label errors?

Confident learning finds label errors by comparing the labels you were given against the labels a model thinks are likely true, using the model’s own uncertainty to decide which disagreements to trust. Rather than picking one probability threshold, it estimates the joint distribution between noisy and true labels and sets a per-class confidence threshold from the data itself (Northcutt, Jiang & Chuang, 2021). The items where a high-confidence prediction contradicts the recorded label rise to the top of the review queue.

The practical payoff is a ranked list. You do not audit the whole dataset; you audit the most likely errors first, which is where the model-assisted labeling mindset of “let the model triage, let a human decide” pays off again. The open-source cleanlab library implements confident learning and produces exactly this ranking of suspected label errors (Cleanlab, 2021), which is how the benchmark study generated its candidates in the first place.

That 51% number is the honest caveat of the whole approach. Confident learning is very good at generating candidates and useless as a final judge. Roughly half of what it flags is not an error, so a pipeline that auto-applies its corrections would inject as much noise as it removes. The output is a shortlist for humans, not a patch to merge.

A worked example: catching a mislabel in a coded set

Here is a synthetic illustration—invented spans, round numbers, no real transcript data—to show the three signals working together on a coded interview set. Suppose you are tagging speech for formal thought disorder using the TLC scale for thought, language and communication, which gives each item an operational definition with verbatim examples (Andreasen, 1986), and tagging depressive content with the MADRS depression scale, a ten-item clinician-rated measure designed to be sensitive to change (Montgomery & Åsberg, 1979).

Take two synthetic spans coded by different annotators:

Span A (coded Derailment): “I need to fix the car, and cars, well, my brother, he never calls on Sundays anymore.”

Span B (coded Tangentiality): “I should fix the car, and cars, you know, my brother, he stopped calling on weekends.”

Both coders were confident. But the two spans are near-duplicates that received different TLC labels—signal three fires immediately. Pull the pair, and the distinction between Derailment and Tangentiality (does the speaker drift mid-thought, or answer obliquely and never return?) has to be resolved once, consistently, for both. That reconciliation is a guideline problem as much as a labeling one, which is why tight annotation guidelines that reduce disagreement prevent these conflicts upstream.

Now scale it up. In a synthetic set of 200 coded spans, imagine two coders disagreed on 24 of them (12%)—signal one—while a confident-learning pass over the same set flagged 30 spans as likely mislabeled—signal two. The two lists overlapped on 11 spans.

Sending all the flagged spans to a third adjudicator resolved 17 as genuine errors and the rest as legitimate ambiguity, a confirmation rate close to the 51% seen in the real benchmark audit.

The lesson from the toy numbers is the one that generalizes: the signals agree often enough to be worth combining and disagree often enough that a human has to make the final call. Disagreement between coders is not noise to be averaged away—handled well, it is the most useful pointer you have, a theme worth reading alongside how Krippendorff’s alpha quantifies that disagreement in the first place.

How do you find label errors in dataset audits?

To find label errors in dataset audits, run a three-stage loop: surface candidates, triage them, and adjudicate the survivors. The detection methods above generate the candidates; the value comes from what you do next, and the annotation-error-detection literature is blunt that methods are rarely compared fairly or evaluated end to end (Klie, Webber & Gurevych, 2023). A ranked candidate list is the start of the work, not the end.

1. Surface. Run all three signals—compute inter-coder disagreement, rank items by confident-learning score, and hash for near-duplicates. Pool the flags into one review queue.
2. Triage. Sort by agreement across signals. Items flagged by two or three methods are the highest-yield; start there. Cap the queue at what your reviewers can clear without rushing, because a rushed re-check just adds a second layer of noise.
3. Adjudicate. Have a qualified reviewer decide each case as error (correct it), ambiguous (leave it, and log why), or guideline gap (fix the scheme so the class stops recurring). Record the decision and the rationale so the audit is reproducible.

Two design choices decide whether this loop helps or hurts. First, how many independent reads each item gets before you trust it—covered in how many annotators medical images need—because a single re-read is itself a single point of failure. Second, whether your adjudicator is deciding against a written definition or against their own memory of one. Confident learning tells you where to look; only a documented adjudication decision tells you what was actually wrong.

Why fixing labels can change which model you pick

Fixing labels matters because label errors do not just lower accuracy uniformly—they can reorder which model looks best. When the benchmark test sets were corrected, the model rankings changed: models that scored highest on the original noisy labels were not always the best on the cleaned labels, and the effect was large enough to flip real model-selection decisions (Northcutt, Athalye & Mueller, 2021). A few percent of label noise, concentrated in the wrong places, is enough to send you home with the wrong model.

This is the argument for treating a dataset-quality audit as a first-class task, not cleanup you do if there is time left. If a 3.3% error rate can change your conclusion about which system is better, then an afternoon spent finding label errors can be worth more than a week of hyperparameter tuning. The same logic applies whenever human labels grade a system, which is why label quality is load-bearing for annotating data for RLHF and LLM evaluation, where noisy preference labels quietly distort the leaderboard.

What automated error detection gets wrong

Automated detection gets three things wrong often enough to plan around. Knowing the failure modes is what separates an audit that improves a dataset from one that scrambles it.

The first is false positives, already quantified: about half of confident-learning flags are not errors (Northcutt, Athalye & Mueller, 2021). Auto-applying corrections would degrade the data. Every flag needs a human verdict.

The second is over-filtering. Filters that aggressively remove suspect items can throw away good data along with bad. Brodley and Friedl showed the trade-off directly: consensus filters (all classifiers must agree an item is noise) are conservative and keep more bad data, while majority filters catch more errors but discard more good data (Brodley & Friedl, 1999). There is no free lunch—only a dial between precision and recall you set on purpose.

The third is mistaking ambiguity for error. Some disagreement reflects genuine boundary cases, not mistakes, and “correcting” a legitimate ambiguity toward one coder’s view manufactures false certainty. This is sharpest on subjective and clinical coding, and it is why the adjudication step must be allowed to return “ambiguous, left as-is” as a valid outcome. If your workflow can only mark things right or wrong, it will silently launder ambiguity into fake ground truth—an especially real risk when weighing synthetic data against human annotation, where model-generated labels look clean but encode the model’s own boundary calls.

Where Tagaroo fits

Tagaroo is a schema-first annotation workspace with review and adjudication built into the workflow: you define a coding scheme once, capture independent reads, see inter-rater reliability, and route disagreements to an adjudicator who records a decision and a rationale. That is the surface where the three-signal audit in this post actually lives—disagreement is measured, an AI first pass can rank candidates, and a qualified human makes the final call. The AI pass is a triage aid that a reviewer signs off on, never a finished label and never a diagnosis.

On data handling, Tagaroo takes a de-identify-first path rather than positioning itself as a processor of identifiable records: its terms require you to strip direct identifiers before upload, its anonymous trial mode runs in the browser so trial text never leaves your machine, and the stack is EU-hosted and GDPR-oriented. It is not a medical device. If your audit must process raw identifiable data in the cloud, put that question to any vendor—including this one—before uploading; see the privacy policy for specifics.

The practical upshot

You will not eliminate label errors, and chasing zero is the wrong goal. The realistic aim is to find label errors in dataset audits efficiently: layer the three signals so their blind spots stop overlapping, let confident learning rank the queue, and spend your scarce expert time adjudicating the top of the list instead of re-reading everything. Half of what the algorithm flags will be fine—that is expected, not a failure of the method.

If you change one thing, change this: stop treating a single label as ground truth and start logging why each disputed label was kept, corrected, or left ambiguous. Then build your coding scheme in Tagaroo with reliability and adjudication in the loop, so the audit is part of labeling instead of an afterthought.

References

• Northcutt CG, Athalye A, Mueller J. Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks. NeurIPS 2021 Datasets and Benchmarks Track. 2021. arXiv:2103.14749 · labelerrors.com
• Northcutt CG, Jiang L, Chuang IL. Confident Learning: Estimating Uncertainty in Dataset Labels. Journal of Artificial Intelligence Research. 2021;70:1373-1411. doi:10.1613/jair.1.12125
• Frénay B, Verleysen M. Classification in the Presence of Label Noise: A Survey. IEEE Transactions on Neural Networks and Learning Systems. 2014;25(5):845-869. doi:10.1109/TNNLS.2013.2292894
• Brodley CE, Friedl MA. Identifying Mislabeled Training Data. Journal of Artificial Intelligence Research. 1999;11:131-167. doi:10.1613/jair.606
• Klie J-C, Webber B, Gurevych I. Annotation Error Detection: Analyzing the Past and Present for a More Coherent Future. Computational Linguistics. 2023;49(1):157-198. doi:10.1162/coli_a_00464
• Cleanlab. cleanlab: open-source library for finding label errors and data-centric AI. 2021. github.com/cleanlab/cleanlab
• Andreasen NC. The Scale for the Assessment of Thought, Language, and Communication (TLC). Schizophrenia Bulletin. 1986;12(3):473-482. doi:10.1093/schbul/12.3.473
• Montgomery SA, Åsberg M. A new depression scale designed to be sensitive to change. British Journal of Psychiatry. 1979;134:382-389. doi:10.1192/bjp.134.4.382