Chest X-ray Annotation: A Worked Example for AI Datasets

Chest X-ray annotation is the process of attaching structured labels to a chest radiograph so a machine-learning model has something correct to learn from: a per-image finding label like cardiomegaly: present, a bounding box around a nodule, or a pixel mask of an effusion. It looks like a labeling chore. It’s really a schema-design problem with a clinical-risk profile, and the datasets that ship clean ground truth win on the decisions made before anyone touches an image.

This is a worked example, not a survey. We’ll take one synthetic frontal film, label it finding by finding, and use each decision to introduce the choices that define a real project: the finding taxonomy, uncertainty labels in the CheXpert style, image-level labels versus region annotation, and the fact that most public labels come from a text-mining tool rather than an expert re-read. It sits inside our medical image annotation guide cluster as the concrete case study.

What is chest X-ray annotation?

Chest X-ray annotation is labeling a chest radiograph with the ground truth a supervised model trains and is tested against. The label can be a whole-image finding class (pneumothorax: yes/no), a bounding box around a lesion, a pixel-level mask of an organ or opacity, or landmark points. The annotation only earns its keep when it’s tied to a defensible notion of truth and to a schema two annotators would read the same way.

Scale sets the tone here. The public datasets that shaped the field label at the whole-image level because that’s what’s affordable at hundreds of thousands of studies.

ChestX-ray8 comprised 108,948 frontal-view images from 32,717 patients with eight text-mined thoracic findings, later expanded to 14 in the widely used ChestX-ray14 release (Wang et al., 2017). CheXpert added 224,316 radiographs from 65,240 patients, each labeled for 14 observations (Irvin et al., 2019). Neither drew a box on the training set.

A worked chest X-ray annotation example, finding by finding

Take one synthetic frontal chest film and read it top to bottom. The point isn’t the radiology—it’s that every finding forces two separate decisions: which schema label applies, and whether a whole-image label is enough or the finding needs a region. Here’s the same film annotated the way a real project would record it.

Finding on the film	Schema label	Certainty	Image-level or region?	Why
Enlarged cardiac silhouette	Cardiomegaly	Positive (1)	Image-level	A global finding; a whole-image label answers 'present?' without a box
Right lower-zone airspace opacity	Consolidation / Pneumonia	Uncertain (-1)	Region (bounding box)	Can't be resolved on one film; box it so a reader or model can revisit the location
Small left apical nodular opacity	Lung nodule	Positive, low confidence	Region (bounding box)	Small and localized—an image-level 'nodule: yes' loses where it is; the box carries location
Blunted left costophrenic angle	Pleural effusion	Positive (1)	Image-level (box if measured)	Presence is clear; a box adds little unless you quantify it
Lungs expanded, no visible pleural air	Pneumothorax	Negative (0)	Image-level	An explicit negative is a label too—'not mentioned' and 'ruled out' are different states

Three lessons fall out of one film. Cardiomegaly needs no box because the label is the answer. The right-lower-zone opacity is genuinely ambiguous, so it earns both an uncertain label and a region, because “we saw something here we couldn’t call” is more useful than a forced guess. And the negative for pneumothorax matters: a schema that can’t distinguish ruled out from never assessed will teach the model the wrong thing.

What label schema should you use for chest findings?

Start from an established finding taxonomy rather than inventing one. The three canonical CXR datasets converged on a short list of thoracic observations, and CheXpert deliberately aligned its terms to the Fleischner Society glossary—the standardized vocabulary for thoracic imaging that defines what “consolidation” or “nodule” means (Hansell et al., 2008; Irvin et al., 2019). Borrowing that vocabulary means your labels bind to terms the wider field already agrees on.

Dataset (year)	Findings	How labels were made	Uncertainty	Region labels?
ChestX-ray8 → ChestX-ray14 (2017)	8, later 14 thoracic findings	NLP text-mining of reports (DNorm + MetaMap), negation removed	Uncertain mentions dropped at labeling	983 images hand-boxed for a localization test; training is image-level
CheXpert (2019)	14 observations, Fleischner-aligned	Rule-based report labeler: extract → classify → aggregate	Explicit positive / negative / uncertain (1 / 0 / -1)	No; image-level only
MIMIC-CXR (2019)	14 (CheXpert labeler applied)	Free-text reports + automated labelers	Inherits CheXpert's uncertain class	No; images + reports, no boxes

The schema is the part that makes agreement possible, and it deserves its own design pass—covered in depth in image labeling taxonomy design. Keep the list short enough to be prevalent and clinically relevant, define each term against the glossary, and version it, because a taxonomy that shifts mid-project silently corrupts everything labeled before the change.

How do you handle uncertain findings?

Treat uncertainty as a label, not a gap. A chest radiograph frequently supports a hedge—“possible early consolidation”, “cannot exclude a small effusion”—and a schema that only offers yes/no forces the annotator to launder that doubt into false confidence. CheXpert’s answer was a third state: each observation is positive (1), negative (0), or uncertain (-1), with a blank reserved for findings the report never mentions (Irvin et al., 2019).

What you do with the uncertain class at training time is a modeling decision, and it isn’t obvious. CheXpert benchmarked five policies and found the winner depended on the finding—so there’s no default to copy blindly.

Policy	What it does with an uncertain label	Where it won in CheXpert
U-Ignore	Drops uncertain instances from training	Baseline; discards signal
U-Zeros	Maps uncertain → negative (0)	Findings where a hedge usually means 'probably absent'
U-Ones	Maps uncertain → positive (1)	Atelectasis and Edema
U-SelfTrained	Train ignoring uncertains, then relabel them with model predictions	Consolidation
U-MultiClass	Treats uncertain as its own third class	Cardiomegaly and Pleural Effusion

The annotation lesson is upstream of the modeling one: capture the uncertainty faithfully at labeling time. If your schema collapses “cannot exclude” into a negative before the data ever reaches a model, you’ve thrown away the exact signal these policies exist to exploit.

Image-level labels or region annotation—which do you need?

Annotate at the coarsest level your task allows. A whole-image finding label answers “is this present?” and is precisely what trained both ChestX-ray8 and CheXpert at scale (Wang et al., 2017; Irvin et al., 2019). Bounding boxes and pixel masks cost far more and introduce disagreement on every boundary, so you buy that precision only when the task genuinely needs location or size.

ChestX-ray8 is the cleanest illustration of the split. Training ran on image-level labels for all 108,948 images, but for evaluating whether the model could localize disease, a board-certified radiologist hand-drew bounding boxes on a small subset—about 983 images, 200 instances per pathology (Wang et al., 2017). Image-level for scale, region-level for the questions that need coordinates.

The rule of thumb: classification for triage and presence, detection when you must point at the finding, segmentation when you must measure it. The full type-by-type treatment lives in image annotation types explained, and once you’re drawing regions, agreement is measured with overlap metrics rather than kappa—see image-segmentation agreement with Dice, IoU, and STAPLE.

Are NLP-derived labels as good as expert reads?

No—and it’s the most important caveat in CXR datasets. The labels in ChestX-ray8, CheXpert, and MIMIC-CXR were produced by automated tools that read the free-text radiology report, not by an expert re-reading each image (Wang et al., 2017; Irvin et al., 2019; Johnson et al., 2019). ChestX-ray8 mined reports with DNorm and MetaMap; CheXpert ran a three-stage rule-based labeler over the report’s Impression section (Wang et al., 2017; Irvin et al., 2019).

That design is what makes hundreds of thousands of labeled studies possible, and it’s a defensible choice—but the label describes the report, not the pixels. It inherits whatever the reporting radiologist hedged, omitted, or phrased unusually, and a labeler mistake on negation can flip a finding. This is why the teams that built these datasets still validated on radiologist-read reference sets: CheXpert’s test set was annotated by eight board-certified radiologists, with the majority vote of five serving as ground truth (Irvin et al., 2019).

How much do radiologists agree on a chest X-ray?

Less than newcomers expect, and that disagreement sets a ceiling on the accuracy you can even measure. Chest radiographs are interpreted, not read off, so two qualified readers will diverge on subtle or boundary-sensitive findings—which is exactly why the reference datasets pool several radiologists rather than trusting one (Irvin et al., 2019).

A concrete measurement makes this tangible. In a study assessing inter-annotator agreement on chest X-rays, two radiologists independently segmented 336 tuberculosis films across 19 abnormality types; the authors quantified agreement with Intersection-over-Union and a STAPLE consensus, and extended Cohen’s and Fleiss’ kappa from categorical labels to pixel-wise segmentation (Yang et al., 2023). The takeaway isn’t a single number—it’s that agreement has to be measured, per finding, and reported alongside the labels.

Two operational habits follow. Combine multiple independent reads into consensus rather than trusting one annotator, and treat recurring disagreement as a signal that your schema is ambiguous, not that a rater is careless. How many readers you actually need is its own question, worked through in how many annotators for medical images.

How do you de-identify chest X-rays before annotation?

De-identify before anyone annotates, and clean two separate places. The first is the DICOM header, where names, IDs, dates, and device details live. The second is the pixel data itself—text burned into the image on secondary captures or scanned films—which header scrubbing never touches. Getting only the header is the most common privacy mistake in imaging datasets.

The public releases model the standard. MIMIC-CXR—377,110 images from 227,835 studies for 65,379 patients—was published only after de-identification to the US HIPAA Safe Harbor standard, with protected health information removed and replaced by placeholder characters, and it remains access-controlled behind a credentialing process and a data use agreement on PhysioNet (Johnson et al., 2019). Because automated pixel cleaning is imperfect, manual inspection of images before release is standard practice (Willemink et al., 2020). The header-and-pixel mechanics, and the DICOM tags involved, are covered in DICOM annotation and data prep.

Where Tagaroo fits

Tagaroo is a schema-first annotation workspace: you define a finding schema once—each label named, defined, and anchored to evidence—then tag against it, with inter-rater reliability and adjudication built into the workflow rather than bolted on afterward. The bet is that CXR projects live or die on the schema and the agreement mechanics, which is where a CheXpert-style label set and a validated clinical rating scale share a spine.

That spine is worth borrowing. A validated rating scale is a curated finding schema with anchored definitions and a citation trail—look at how the MADRS specifies each depression item, or how the PHQ-9 fixes its nine items and scoring. A chest-finding taxonomy should aim for the same bar: every term defined against the Fleischner glossary, every ambiguity pre-resolved, the whole scheme versioned. When your labels also span reports and images together, the same engine handles both—see multimodal image and text annotation.

On data handling, Tagaroo takes the de-identify-first path this example argues for rather than positioning itself as a PHI processor: its terms require you to strip direct identifiers—DICOM header fields and burned-in pixel PHI—before upload, its anonymous trial mode runs entirely in the browser so trial images never leave your machine, and the stack is EU-hosted and GDPR-oriented. It is not a medical device, and any AI pre-label is a first pass for a qualified human to review, never a diagnosis. If your project must process raw PHI in the cloud under a signed Business Associate Agreement, put that question to any vendor—including this one—before uploading identifiable studies; see Tagaroo’s privacy policy for specifics.

The practical upshot

Good chest X-ray annotation is schema design first and drawing second. Anchor your finding taxonomy to a standard glossary, make uncertainty a real label instead of a forced guess, annotate at the coarsest level your task allows and add regions only where location or size matters, and never confuse a report-derived label with an image read. Do that, and the label becomes what it was always meant to be: trustworthy ground truth you can defend.

Start from the schema, because it decides how much you’ll fight later. If you want a workspace that treats the finding schema and inter-rater reliability as first-class from the first label, build your annotation schema in Tagaroo—and use the cluster guides above for the parts that need the math.

References

• Irvin J, Rajpurkar P, Ko M, et al. CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison. Proceedings of the AAAI Conference on Artificial Intelligence. 2019;33(01):590–597. doi:10.1609/aaai.v33i01.3301590 (arXiv:1901.07031)
• Wang X, Peng Y, Lu L, Lu Z, Bagheri M, Summers RM. ChestX-ray8: Hospital-Scale Chest X-Ray Database and Benchmarks on Weakly-Supervised Classification and Localization of Common Thorax Diseases. IEEE CVPR. 2017:2097–2106. arxiv.org/abs/1705.02315
• Johnson AEW, Pollard TJ, Berkowitz SJ, et al. MIMIC-CXR, a de-identified publicly available database of chest radiographs with free-text reports. Scientific Data. 2019;6:317. doi:10.1038/s41597-019-0322-0
• Yang F, Zamzmi G, Angara S, et al. Assessing Inter-Annotator Agreement for Medical Image Segmentation. IEEE Access. 2023;11. doi:10.1109/ACCESS.2023.3249759
• Hansell DM, Bankier AA, MacMahon H, McLoud TC, Müller NL, Remy J. Fleischner Society: Glossary of Terms for Thoracic Imaging. Radiology. 2008;246(3):697–722. doi:10.1148/radiol.2462070712
• Willemink MJ, Koszek WA, Hardell C, et al. Preparing Medical Imaging Data for Machine Learning. Radiology. 2020;295(1):4–15. doi:10.1148/radiol.2020192224