Data Cascades: Why Fixing Labels Beats Tuning Models

Data cascades are compounding downstream failures that trace back to upstream data problems, and they are close to universal. In a study of 53 practitioners building high-stakes AI systems—landslide detection, suicide prevention, cancer detection—92% reported hitting at least one, and 45.3% reported two or more in a single project (Sambasivan et al., 2021). The uncomfortable part is where the effort usually goes: teams pour weeks into the model while the damage was set in motion earlier, in how the data was defined and labeled.

This post argues that fixing your labels usually beats tuning your model, and shows what a data-centric practice looks like in place of the reflex to reach for a bigger architecture. It covers what a data cascade actually is, why upstream problems compound instead of staying local, the model-centric-versus-data-centric split, a worked synthetic example, and where the argument has limits.

What is a data cascade?

A data cascade is a compounding sequence of downstream problems caused by an upstream data issue, producing negative effects that surface far from where they began (Sambasivan et al., 2021). The term comes from a Google study led by Nithya Sambasivan, who interviewed 53 practitioners deploying AI in high-stakes settings and found the pattern almost everywhere they looked.

The number that makes the paper worth citing is the prevalence. Cascades were not an edge case but the default: 92% of practitioners reported at least one, and nearly half reported two or more in the same project (Sambasivan et al., 2021). When something happens to nine teams in ten, it stops being bad luck and starts being a property of how the work is structured.

What makes a cascade distinct from ordinary noise is the delay between cause and cost. The paper’s phrase for it is memorable—teams treat data as “operations,” something to get past on the way to the interesting modeling work—and the neglect is invisible until a downstream metric or a deployed decision goes wrong. By then the trail back to the ambiguous label or the missing category is cold.

Why do upstream data problems compound downstream?

Upstream data problems compound because every stage after labeling inherits the labels as ground truth and builds on top of them without re-checking. A wrong or ambiguous label is not corrected by training; it is amplified—the model learns the error as if it were signal, and evaluation confirms the model because the test set carries the same flaw.

This is the sense in which “garbage in, garbage out” undersells the problem. That phrase describes a one-step failure. A cascade is multi-step: the defect moves through the guideline, the annotation, the training set, the held-out set, and the deployment, and at each hop it can be laundered into something that looks clean. Label noise has many ordinary causes—unclear instructions, genuine expert mistakes, subjective categories, rushed work at scale—and none of them require carelessness to produce a residue of wrong labels (Frénay & Verleysen, 2014).

The evaluation trap is the cruel part. If your test labels were produced by the same process as your training labels, a shared bias will not show up as lower accuracy. The metric agrees with the model because both are wrong in the same direction, which is exactly why teams can ship a confident system that fails on contact with reality.

Model-centric vs data-centric AI: what’s the difference?

Model-centric development holds the dataset fixed and iterates on the model; data-centric development holds the model fixed and iterates on the data. Andrew Ng, who coined the term in 2021, defines data-centric AI as “the discipline of systematically engineering the data used to build an AI system” (Ng, 2021). The reframing matters because the two approaches fight different failures and have different payoff curves.

The field spent a decade optimizing models against frozen benchmarks, which quietly taught everyone that data was a settled input rather than the main lever. DataPerf pushes the correction into infrastructure: it benchmarks datasets and data-cleaning algorithms directly, on the argument that neglecting data has produced inaccuracy, bias, and fragility in real systems (Mazumder et al., 2023). When the data is the bottleneck—and with 92% cascade prevalence, it usually is—model tuning hits diminishing returns while data work does not.

Dimension	Model-centric	Data-centric
What you hold fixed	The dataset; you iterate on architecture and hyperparameters	The model; you iterate on data quality and coverage (Ng, 2021)
Where effort goes	Architecture search, ensembling, tuning	Guidelines, label review, adjudication, coverage gaps
Primary failure fought	Underfitting and overfitting	Data cascades: compounding upstream errors (Sambasivan et al., 2021)
Payoff when labels are noisy	Diminishing once data is the limit	Often the larger remaining gain
How progress is benchmarked	Leaderboard accuracy on a fixed set	DataPerf scores the dataset, not just the model (Mazumder et al., 2023)

This is not an argument to stop tuning models. It is an argument about sequence: when a few percent of your labels are wrong in correlated ways, the cheapest large improvement available is upstream, not in the optimizer.

Where do data cascades start?

Data cascades start at the points where meaning gets fixed: the coding scheme, the annotator’s judgment, and the label itself. Sambasivan and colleagues traced cascades to concrete triggers—physical-world brittleness, inadequate domain expertise on the labeling side, conflicting reward structures, and poor cross-team documentation (Sambasivan et al., 2021). Every one of those is a decision made before a model is trained.

The most controllable trigger is definitional. A category that two qualified coders interpret differently will produce inconsistent labels no matter how careful the coders are, which is why annotation guidelines that reduce disagreement are the first line of defense, and why taxonomy design for image labeling is a data-quality decision rather than a formatting one. The second controllable trigger is disagreement you cannot see. If you never measure inter-coder reliability, you cannot tell a real boundary case from a coder who misread the guideline—a distinction worth reading about in why disagreement is signal, not noise.

There is also an incentive layer. Cascades thrive where the people defining and labeling data are rushed, undervalued, or paid too little to care about edge cases, which is one reason whether pay improves annotation quality is a real question and not a soft one. The upstream fixes are unglamorous, and that is precisely why they get skipped.

A worked example: how one label decision cascades

Here is a synthetic illustration—invented spans, round numbers, no real transcript data—to show a cascade forming from a single undefined boundary. Suppose you are tagging interview speech for formal thought disorder with the TLC scale for thought, language and communication, Andreasen’s glossary of operationally defined speech disorders (Andreasen, 1986), and tagging depressive content with the MADRS depression scale, a ten-item clinician-rated measure designed to be sensitive to change (Montgomery & Åsberg, 1979).

The guideline never pins down Derailment (drifting off track mid-thought) versus Tangentiality (answering obliquely and never returning). Two coders tag 300 spans; they split on 18% of the boundary cases without either noticing, and the labels are merged by majority vote. So far this looks like an ordinary annotation project.

Now the cascade. A classifier trained on the merged set reaches 0.91 accuracy on a test split drawn from the same labels—and learns the coders’ confusion as if it were the real Derailment-Tangentiality boundary. Because the held-out labels carry the same 18% ambiguity, the metric looks healthy while the model has quietly encoded a distinction nobody actually agreed on. One undefined boundary upstream became a confident-but-wrong model downstream, plus a number that hides it.

The fix is not a better optimizer. It is resolving the boundary once, in the guideline, and re-checking the disputed spans—the same layered audit described in how to find label errors in your dataset. Measured properly, that 18% disagreement was the most useful diagnostic in the whole project; averaged away by majority vote, it became invisible debt.

Why does fixing labels beat tuning the model?

Fixing labels beats tuning the model because label errors do not lower accuracy uniformly—they can reorder which model looks best, so cleaning them changes your decision, not just your score. When researchers corrected the label errors in ten of the most-cited machine-learning benchmarks, the model rankings shifted: systems that scored highest on the original noisy labels were not always the best on the cleaned labels (Northcutt, Athalye & Mueller, 2021). The same study found an average of 3.3% label errors across those benchmarks, rising to about 6% of the ImageNet validation set.

Sit with the implication. These are the datasets the field trusts most, and a few percent of concentrated label noise was enough to flip real model-selection decisions. If that can happen to ImageNet, the dataset your team labeled in six weeks under deadline is not cleaner by accident, and the model you picked from it may be the wrong one for a reason that no amount of tuning will reveal.

That is the argument in one line: when a 3.3% error rate can change your conclusion about which system is better, an afternoon spent auditing labels can outperform a week of hyperparameter search. It generalizes anywhere human labels grade a system—including the preference labels behind RLHF and synthetic-versus-human annotation trade-offs, where clean-looking model-generated labels quietly encode the generator’s own boundary calls.

How do you work data-centrically without over-correcting?

You work data-centrically by fixing definitions and measuring agreement first, then treating automated error-detection as a ranked queue for human review rather than a patch to auto-apply. The order matters: cleaning labels with a filter before your guideline is settled just launders one coder’s opinion into fake ground truth.

Three cautions keep a data-centric push honest. First, do not chase zero errors—some disagreement is legitimate ambiguity, and “correcting” a real boundary case toward one coder’s view manufactures false certainty rather than removing noise. Second, measure before you clean: without a reliability coefficient like Cohen’s kappa for inter-rater agreement, you cannot separate a fixable error from an honest boundary. Third, remember that automated flagging is a filter, not a verdict—error-detection methods generate candidates, and only a documented human adjudication decides what was actually wrong.

The failure mode of data-centric enthusiasm is over-filtering: aggressive removal of “suspect” items throws away good data along with bad, trading one bias for another. There is no free lunch here, only a precision-recall dial you set on purpose. Data-centric practice is not a promise that clean data is easy; it is a claim about where the largest remaining gain sits.

Where Tagaroo fits

Tagaroo is a schema-first annotation workspace built around the upstream half of this argument: you define a coding scheme once, capture independent reads, see inter-rater reliability instead of averaging it away, and route disagreements to an adjudicator who records a decision and a rationale. That is the surface where data cascades are caught early—at the guideline and the label—rather than surfaced late from a deployment metric. An AI first pass can rank suspect labels, but a qualified human signs off on every one; the AI is a triage aid, never a finished label and never a diagnosis.

On data handling, Tagaroo takes a de-identify-first path: its terms require you to strip direct identifiers before upload, its anonymous trial mode runs in the browser so trial text never leaves your machine, and the stack is EU-hosted and GDPR-oriented. It is not a medical device. If your workflow must process raw identifiable data in the cloud, put that question to any vendor—including this one—before uploading; see the privacy policy for specifics.

The practical upshot

Data cascades are a structural feature of how AI gets built, not a run of bad luck: nine teams in ten hit at least one, and they cost the most precisely because the bill arrives far downstream of the cause (Sambasivan et al., 2021). The reflex to answer a data problem with a bigger model is understandable and, when labels are noisy, mostly wasted.

If you change one thing, change the sequence. Settle the definitions, measure the disagreement, and adjudicate the disputes before you touch the optimizer—then build your coding scheme in Tagaroo with reliability and adjudication in the loop, so the audit is part of labeling instead of an afterthought. Fix the data work everyone would rather skip, and the model work gets easier on its own.

References

• Sambasivan N, Kapania S, Highfill H, Akrong D, Paritosh P, Aroyo LM. “Everyone wants to do the model work, not the data work”: Data Cascades in High-Stakes AI. Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. 2021. doi:10.1145/3411764.3445518
• Ng A. From Model-Centric to Data-Centric AI. DeepLearning.AI campaign, 2021. Forbes coverage
• Mazumder M, Banbury C, Yao X, et al. DataPerf: Benchmarks for Data-Centric AI Development. Advances in Neural Information Processing Systems 36 (NeurIPS 2023). 2023. arXiv:2207.10062
• Northcutt CG, Athalye A, Mueller J. Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks. NeurIPS 2021 Datasets and Benchmarks Track. 2021. arXiv:2103.14749 · labelerrors.com
• Frénay B, Verleysen M. Classification in the Presence of Label Noise: A Survey. IEEE Transactions on Neural Networks and Learning Systems. 2014;25(5):845-869. doi:10.1109/TNNLS.2013.2292894
• Andreasen NC. The Scale for the Assessment of Thought, Language, and Communication (TLC). Schizophrenia Bulletin. 1986;12(3):473-482. doi:10.1093/schbul/12.3.473
• Montgomery SA, Åsberg M. A new depression scale designed to be sensitive to change. British Journal of Psychiatry. 1979;134:382-389. doi:10.1192/bjp.134.4.382