Ground Truth for Medical AI: Reference Standards & the FDA

Ground truth for medical AI is a promise the phrase mostly can’t keep. For most clinical imaging tasks there is no single, knowable truth to label against, only a carefully constructed best estimate built from several experts and a documented process. That gap between the truth you wish you had and the reference standard you can actually build is where FDA reviewers spend their attention, and where a surprising number of medical-AI datasets quietly go wrong.

Here’s the shape of it, up front. When you annotate images to support a computer-aided detection (CADe) or computer-aided diagnosis (CADx) device, what you are really building is a reference standard: an operational definition of truth, set by a blinded expert panel, adjudicated where readers disagree, and then evaluated with a multi-reader multi-case (MRMC) study. This post works through the reference-standard-versus-ground-truth distinction, expert-panel construction, MRMC design, the FDA’s CADe performance thinking, and the SegAgree interchangeability idea, with a synthetic worked example you can reason about.

Reference standard vs ground truth: what’s the difference?

A reference standard and ground truth are not the same thing, and the difference is not pedantic. Ground truth is the actual underlying disease state, which is frequently unknowable at labeling time. A reference standard is the best available operational proxy for it: the label a defined process produces, such as the majority read of a blinded expert panel, an adjudicated consensus, or a definitive test like biopsy or long-term follow-up.

FDA’s CADe guidance is careful about this. It describes a “truthing process” that establishes a “truth status” for each case, then a separate “scoring process” that decides whether a reader’s mark corresponds to that truth (FDA, 2022). Notice the framing: truth is something you establish through a process, not something you read off the image. The honest version of “ground truth for medical AI” is almost always a reference standard wearing the word’s clothes.

Why does the wording matter for a dataset? Because it changes what you must document. If you claim ground truth, a reviewer expects certainty you rarely have.

If you instead claim a reference standard, you owe a description of the process that produced it: who read the cases, how they were blinded, how disagreements were resolved, and what residual uncertainty remains. That documentation is the difference between a defensible dataset and a black box.

What counts as ground truth for medical AI?

For a device submission, ground truth for medical AI counts only when it comes from a pre-specified, documented reference-standard process, not from a single reader’s opinion. One read is one clinician’s judgment, complete with that clinician’s blind spots. The reference standard is what you get after combining several independent reads and resolving the conflicts on purpose.

The most persuasive datasets make this explicit. In the diabetic-retinopathy deep-learning study, the algorithm was not validated against any one grader; the reference standard was the majority decision of a panel of US board-certified ophthalmologists (Gulshan et al., 2016). The single-reader label was treated as an input to a process, not as truth. That is the pattern to copy: define truth as the output of a panel and an adjudication rule, and write it down before labeling starts.

The reference-standard design you choose should track how much a wrong label costs downstream. The table below lays out the common options, from the cheapest to the most defensible, with the weakness each one carries.

Reference-standard design	How truth is set	Fits	Main weakness
Single expert read	One reader's label taken as truth	Low-risk pre-labels, triage sorting	Carries one reader's bias; not truth for anything consequential
Blinded panel + majority	Independent blinded reads combined by majority vote	Detection and classification labels	Vote is meaningless if reads are not blind and independent
Panel + adjudication	Panel reads, a senior reader resolves the splits	Diagnostic or boundary-sensitive findings	A residual share of disagreement is irreducible
Definitive outcome proof	Biopsy, histopathology, or follow-up outcome	Cancer confirmation, when it is obtainable	Often unavailable, delayed, or invasive to obtain
No reference standard (interchangeability)	Compare device-to-expert vs expert-to-expert disagreement	Segmentation where no gold standard exists	Overlap-based only; treats the reader effect as fixed

How do you build an expert-panel reference standard?

You build it by fixing the process before anyone reads: how many experts, how they are blinded, how their reads are combined, and how disagreements are adjudicated. The order matters. Deciding the combination rule after you have seen the disagreements is how bias creeps in.

Three design choices carry most of the weight. First, the reads must be blind and independent, so that reader errors stay uncorrelated and a majority vote actually means something. Second, you need a combination rule matched to the output: a majority vote for categorical findings, the median for measurements, and a statistical fusion method for segmentation masks. Third, you need an adjudication step where a senior reader resolves only the cases the rule flagged as discordant, rather than re-reading everything.

How many experts is its own question, and it has no single answer; it scales with clinical risk and with what the label feeds. We work through read counts, blinding, and adjudication in detail in how many annotators medical images need, and the spatial-agreement mechanics for masks in image segmentation agreement with Dice, IoU, and STAPLE. The construction question is the same whether your labels are pixels or diagnostic categories; a diagnostic taxonomy built from criteria is covered in turning DSM criteria into annotation schemes.

What is a multi-reader multi-case (MRMC) study?

A multi-reader multi-case (MRMC) study is the standard reader-study design for evaluating diagnostic imaging AI: several clinicians each interpret the same set of patient cases under two or more conditions, usually without the device and with it, so the analysis can separate three sources of variation at once: differences between readers, differences between cases, and the effect of the device (Gallas et al., 2012). The most common form is “fully crossed,” where every reader reads every case in every condition.

The endpoints are usually built on the ROC curve. FDA’s CADe guidance names ROC-based metrics, and variants like the free-response ROC (FROC) and location-specific ROC (LROC), alongside sensitivity and specificity at a clinical action point, as the likely candidates for study endpoints (FDA, 2022). The FROC variant matters for detection tasks because it accounts for multiple findings per image and where they sit, not just whether the image is abnormal.

Sizing an MRMC study is where teams underestimate the cost. The variance of the estimate depends on reader variability, case variability, and the correlations between reads, so you cannot pick a reader count off a chart (Gallas et al., 2012; Gallas, 2006). The FDA-MIPS workshop that produced much of this methodology, with statisticians including Nancy Obuchowski, reached a working consensus that a study may need at least 10 readers to keep reader variability from swamping statistical power for an FDA submission (Gallas et al., 2012).

Standalone vs clinical performance in FDA’s CADe thinking

FDA distinguishes two performance questions, and a submission often has to answer both. Standalone performance measures how well the device marks known abnormalities and avoids marking normal tissue, entirely on its own. Clinical performance measures whether clinicians do better with the device than without it, which is the MRMC reader study (FDA, 2022).

The guidance is explicit that these are different studies with different scoring. Standalone scoring compares the device’s marks to the reference standard directly; clinical scoring determines whether a reader’s interpretation corresponds to the established truth status, and the guidance devotes a full section to the reference standard and the truthing process behind it (FDA, 2022). There are narrow situations, such as a direct comparison of a modified device to its own prior version, where a new clinical study may not be needed, but FDA notes that indirect comparisons across different datasets and reference standards are hard to defend (FDA, 2022).

For a data team, the practical reading is that your reference standard is not a labeling detail; it is a load-bearing part of both study types. Two submissions with different truthing processes are not directly comparable, which is exactly why the process has to be pre-specified and documented.

The SegAgree idea: interchangeability without a reference standard

SegAgree is an FDA regulatory-science tool that turns a hard question, “is this segmentation good enough?”, into a comparison you can actually make when there is no gold standard. Instead of scoring the device against one aggregated reference, it compares how much the device disagrees with each expert to how much the experts disagree with each other, using pairwise Dice scores, and reports the mean Dice difference with a 95% confidence interval (FDA, 2026; Hu et al., 2025).

The FDA describes the tool as implementing “a statistical segmentation interchangeability framework that directly compares device-to-expert dissimilarity with expert-to-expert dissimilarity” and quantifying agreement “without requiring a reference standard” (FDA, 2026). The underlying idea is the one worth memorizing: a device can be judged acceptable when it differs from the expert panel no more than the panel members differ among themselves. If experts routinely disagree on where a lesion’s boundary is, a device that lands inside that spread of human disagreement is, in a meaningful sense, interchangeable with a human reader.

The method was validated with statistical and image-based simulations and applied to an AI segmentation algorithm against four radiologists using data from the Lung Image Database Consortium, controlling the false-positive (Type I) error rate near the nominal 0.05 when device and experts genuinely agreed (Hu et al., 2025). It has real limits, which the authors state plainly: it is scoped to overlap-based segmentation performance only, and it treats the reader effect as fixed rather than random (FDA, 2026; Hu et al., 2025). It is not a general substitute for a reference standard; it is a principled answer for the case where one cannot honestly be built.

A synthetic worked example

Here is a small, deliberately synthetic example to make interchangeability concrete. Suppose four radiologists each segment the same lung nodule, and you compute the Dice overlap for every pair of experts. Say those expert-to-expert Dice scores average 0.82, with pairs ranging from about 0.74 to 0.89. These are illustrative numbers, not measured data.

Now bring in the device. Compute the Dice overlap between the device’s mask and each of the four experts, and average it. If that device-to-expert average is 0.80, the mean Dice difference between the two comparisons is small, roughly 0.02, and if its 95% confidence interval comfortably includes zero, the device sits inside the human spread. The reading is not “the device scored 0.80, is that a pass?”, a threshold nobody can defend, but “the device disagrees with the panel about as much as the panel disagrees with itself.” That is the shift SegAgree encodes (FDA, 2026; Hu et al., 2025).

Contrast that with the naive approach: fuse the four experts into one consensus mask, score the device against it, and announce Dice 0.80 as if the number were self-interpreting. The consensus hides the human disagreement that gives 0.80 its meaning. If you must use overlap thresholds, understand what they do and do not capture; we cover that in Dice vs IoU vs Hausdorff.

Limitations and where teams get this wrong

The most common mistake is treating a single expert’s labels as ground truth for medical AI and skipping the process entirely. A single read is an opinion, and a dataset built on one is not defensible for anything that informs care. The fix is not a bigger model; it is a documented reference-standard process.

The second mistake is unblinded consensus. If a second reader can see the first read, agreement inflates and the independence that makes a vote meaningful is gone. The third is assuming adjudication converges everything. A residual share of disagreement is irreducible, reflecting genuine ambiguity in the image rather than a careless reader, and forcing it to a single label manufactures false precision; that residual belongs in your uncertainty budget.

A subtler trap is over-reading what FDA guidance says. The guidance describes recommendations and evaluation paradigms; it does not hand you a fixed recipe that by itself secures clearance, and it repeatedly points you to discuss your specific device with the Agency (FDA, 2022). Anyone who tells you a dataset is “FDA-compliant” as a property of the labels is overselling. Compliance is a property of a submission and its review, not of an annotation file.

How curated coding schemes reduce disagreement

The cheapest way to need less adjudication is to remove the disagreement you can before anyone reads, with a precise, anchored coding scheme. A well-specified definition converts a chunk of resolvable disagreement into simple agreement: if two readers apply the same operational definition the same way, there is no tie to break.

Clinical measurement solved a version of this decades ago, and imaging teams can borrow the pattern. A validated rating scale is a curated coding scheme with anchored definitions, a fixed structure, and a citation trail. The MADRS depression scale, a ten-item clinician-rated instrument designed to be sensitive to change (Montgomery & Åsberg, 1979), gives every item severity anchors, and the self-report PHQ-9, a nine-item measure (Kroenke, Spitzer & Williams, 2001), fixes exactly what each item asks.

That anchoring is the bar an image-annotation data dictionary should aim for: every label defined, every foreseeable ambiguity pre-resolved. The same discipline underlies whole-slide pathology work, where region definitions have to be pinned down before scale-out labeling; see whole-slide image annotation for pathology AI.

Where Tagaroo fits, and where it doesn’t

Tagaroo is a schema-first annotation workspace: you define a coding scheme once, then tag against it with inter-rater reliability and adjudication built into the workflow rather than bolted on afterward. Blind parallel reads, a pre-specified combination rule, and agreement computed as annotators work are the parts that decide whether a multi-reader design produces a defensible reference standard, and they are the parts general-purpose tools tend to leave to a spreadsheet.

Be clear about the boundary. If your job is industrial-scale voxel segmentation of DICOM volumes with specialized auto-annotation, a dedicated computer-vision platform is the right call, and the de-identification mechanics for that data live in preparing DICOM data for annotation, with a concrete read-through in our chest X-ray annotation example. Tagaroo’s strength is guided, schema-driven annotation with every label anchored to the evidence that justifies it. For the broader picture of building imaging datasets, start with our medical image annotation guide.

On data handling, Tagaroo takes the de-identify-first path this guide argues for, rather than positioning itself as a PHI processor. Its terms require you to strip direct identifiers, DICOM header fields and burned-in pixel PHI, before upload; its anonymous trial mode runs entirely in the browser so trial images never leave your machine; and the stack is EU-hosted and GDPR-oriented. Tagaroo is not a medical device, is not FDA-cleared, and does not produce FDA-cleared output; any AI pre-label is a first pass for a qualified human to review, never a diagnosis. If your project must process raw PHI in the cloud under a signed Business Associate Agreement, put that question to any vendor, including this one, before uploading identifiable studies; see Tagaroo’s privacy policy for specifics.

The practical upshot

Ground truth for medical AI is almost always a reference standard: a documented process, not a fact you read off the pixels. Build it from a blinded expert panel, combine reads with a rule you fix in advance, adjudicate the splits, and evaluate the device with an MRMC study using ROC- or FROC-based endpoints (FDA, 2022; Gallas et al., 2012). When no gold standard can honestly be built, the interchangeability framing behind SegAgree, device disagreeing with experts no more than experts disagree with each other, is the principled way to ask whether a segmentation is good enough (FDA, 2026; Hu et al., 2025).

If you change one thing, change this: stop calling one reader’s labels ground truth. Start from the anchored coding scheme that makes agreement possible, and build your reference-standard scheme in Tagaroo with reliability and adjudication in the loop from the first label.

References

• U.S. Food and Drug Administration (CDRH). Clinical Performance Assessment: Considerations for Computer-Assisted Detection Devices Applied to Radiology Images and Radiology Device Data in Premarket Notification (510(k)) Submissions. Guidance for Industry and FDA Staff. Issued September 28, 2022. fda.gov
• U.S. Food and Drug Administration (CDRH). Computer-Assisted Detection Devices Applied to Radiology Images and Radiology Device Data - Premarket Notification [510(k)] Submissions. Guidance for Industry and FDA Staff. Issued September 28, 2022. fda.gov
• U.S. Food and Drug Administration (CDRH). SegAgree: Statistical Assessment of Agreement in Overlap-Based Performance Between an AI Segmentation Device and a Multi-Expert Human Panel (RST26AI03.01). Regulatory Science Tools Catalog, 2026. cdrh-rst.fda.gov
• Hu T, Sahiner B, Guan S, Mikailov M, Cha K, Samuelson FW, et al. Statistical testing of agreement in overlap-based performance between an AI segmentation device and a multi-expert human panel without requiring a reference standard. Journal of Medical Imaging. 2025;12(5):055003. doi:10.1117/1.JMI.12.5.055003
• Gallas BD, Chan H-P, D’Orsi CJ, Dodd LE, Giger ML, Gur D, et al. Evaluating Imaging and Computer-aided Detection and Diagnosis Devices at the FDA. Academic Radiology. 2012;19(4):463-477. doi:10.1016/j.acra.2011.12.016
• Gallas BD. One-Shot Estimate of MRMC Variance: AUC. Academic Radiology. 2006;13(3):353-362. doi:10.1016/j.acra.2005.11.030
• Gulshan V, Peng L, Coram M, Stumpe MC, Wu D, Narayanaswamy A, et al. Development and Validation of a Deep Learning Algorithm for Detection of Diabetic Retinopathy in Retinal Fundus Photographs. JAMA. 2016;316(22):2402-2410. doi:10.1001/jama.2016.17216
• Montgomery SA, Åsberg M. A new depression scale designed to be sensitive to change. British Journal of Psychiatry. 1979;134:382-389. doi:10.1192/bjp.134.4.382
• Kroenke K, Spitzer RL, Williams JBW. The PHQ-9: validity of a brief depression severity measure. Journal of General Internal Medicine. 2001;16(9):606-613. doi:10.1046/j.1525-1497.2001.016009606.x