Medical Image Annotation: A Complete Guide for AI Teams

Medical image annotation is the process of labeling medical images—CT, MRI, X-ray, ultrasound, or digitized pathology slides—with the structured ground truth an AI model learns from: a class, a bounding box, a pixel-level mask, or an anatomical landmark. It sounds like a drawing task. In practice it’s a measurement problem with a clinical-risk profile, and the projects that fail usually fail on the parts nobody screenshots: de-identification, who is qualified to label, and how you turn several expert opinions into one defensible answer.

This is the hub for our image-annotation cluster. It walks the whole pipeline once, at altitude, then points you to the deep-dives for the parts that deserve their own article. If you take one idea away, take this: the label is not the deliverable. The deliverable is trustworthy ground truth, and that is a process outcome, not a drawing.

What is medical image annotation?

Medical image annotation is labeling medical images with the ground truth a supervised model trains and is tested against. The label can be a whole-image class (“pneumothorax: yes/no”), a bounding box around a lesion, a pixel-level segmentation mask of an organ or tumor, or landmark points on anatomy. The annotation is only useful if it’s tied to a defensible notion of truth.

Here’s the distinction most guides blur. The term ground truth properly refers to information from direct observation—a biopsy, a lab result, a surgical finding. Image labels are annotations made by experts, and they count as ground truth only when imaging itself is the reference standard, as with pneumothorax, fractures, or intracranial hemorrhage (Willemink et al., 2020).

For most conditions, an image alone doesn’t settle the diagnosis; an opacity on a chest radiograph has a long differential, and the real label depends on pathology, follow-up imaging, or clinical outcome. Deciding what your label actually represents is the first design choice, and it constrains everything downstream.

The medical image annotation workflow, end to end

The dependable medical image annotation workflow is linear, and each stage exists to protect the next one. Read it as a checklist you can lift into a project plan:

1. Collect and govern. Assemble a representative dataset under the required ethics or IRB approval, and define the clinical task and the reference standard before anyone labels. What counts as truth decides who annotates and how you’ll adjudicate.
2. De-identify. Strip protected health information from the DICOM header and from any text burned into the pixels, then inspect images by hand where automation can’t be trusted.
3. Annotate. Apply a written protocol and a data dictionary. For anything that needs consensus, annotators work blind to each other so their opinions stay independent.
4. Adjudicate. Combine multiple annotations into one ground truth—majority vote, median, or STAPLE—escalating only genuine disagreements to a senior reviewer.
5. QA. Measure agreement and completeness, set the human review rate by clinical risk, and log every flagged annotation until it’s resolved.
6. Export. Save annotations in an interoperable schema alongside the data dictionary, so the dataset stays reusable across tools and sites.

Stage	What happens	Who owns it	Where it breaks
Collect & govern	Representative sampling; ethics/IRB; define task + reference standard	PI / data lead	Ground truth undefined until after labeling starts
De-identify	Clean DICOM header + burned-in pixel PHI; manual spot-check	Data engineering + privacy	Pixel PHI missed by automated tools
Annotate	Protocol + data dictionary; blind, independent labeling	Annotators (expert or trained)	No codebook; annotators see each other's work
Adjudicate	Majority / median / STAPLE; senior review on disagreement	Adjudicator / senior clinician	One rater treated as truth; no escalation rule
QA	Agreement + completeness metrics; risk-based review rate	QA lead	Fixed review rate regardless of clinical risk
Export	Interoperable schema + data dictionary shipped together	Data engineering	Proprietary format, no dictionary, not reusable

The order isn’t cosmetic. If you de-identify after annotating, you risk leaking PHI into label files and exports. If you skip a written protocol, your adjudication step inherits disagreements that are really definitional, not clinical. Fix the process and the labels take care of themselves.

What are the types of medical image annotation?

There are four workhorse annotation types, ordered from cheapest to most demanding. This is the at-a-glance version; the full type-by-type treatment, with when-to-use-which, lives in image annotation types explained.

Type	What you produce	Typical use	Relative cost
Classification	One label per image or study	Triage, normal-vs-abnormal, quality checks	Low
Detection	Bounding boxes around findings	Nodule/lesion localization, counting	Medium
Segmentation	Pixel- or voxel-level masks	Volume, margins, organ delineation	High
Landmarks	Keypoints on anatomy	Measurements, registration, morphometry	Medium

The practical rule: annotate at the coarsest level your task allows. If a classification label answers your clinical question, don’t pay for segmentation. Precision you don’t need becomes disagreement you have to adjudicate, and segmentation is where raters diverge most because every boundary pixel is a judgment call.

Who should annotate medical images?

Whether you need a radiologist or a trained non-expert depends entirely on the reference standard your labels have to meet. Expert reads are the ground truth only when imaging is definitive; when it isn’t, no amount of annotator seniority manufactures truth that the image doesn’t contain (Willemink et al., 2020). So the honest question isn’t “expert or not” but “what standard does this label have to satisfy, and who can satisfy it?”

For diagnostic or boundary-sensitive work—tumor margins, subtle findings, ambiguous pathology—you want domain experts, and often more than one. In radiology practice, double reading is common: each image is read by at least two clinicians, and validation sets are frequently labeled by several (Encord, 2022). Computational pathology projects go further and use a multidisciplinary team—pathologists working alongside ML engineers against a shared data dictionary (Wahab et al., 2022).

Trained non-experts have a real place, and it’s larger than experts like to admit. For well-specified, lower-risk tasks—image-quality flags, coarse triage, pre-annotation the expert then corrects—a trained annotator under a clear protocol and tight QA is both faster and cheaper. The move that fails is using non-experts on high-risk labels with no expert anchor.

Match the annotator to the risk, not to the org chart. For a longer treatment of team size and mix, and the tools that support each, see our roundup of the best data annotation tools.

How do you turn multiple annotations into ground truth?

You combine them, because one annotation is an opinion and ground truth is a consensus. A single expert’s segmentation carries that expert’s biases and a bad day; pooling several independent reads is how you get an answer you can defend to a reviewer or a regulator. The mechanics differ by annotation type, and you should pre-specify the rule before labeling starts, not argue it afterward (Innolitics, 2025).

Three patterns cover most projects. For categorical labels, use 2-of-3 majority voting across three blinded readers, with a “2+1” escalation where an adjudicator engages only on disagreement. For measurements (a diameter, a volume), take the median or the two most concordant values, escalating only when readers exceed a pre-set gap.

For segmentation masks, don’t hold meetings—run STAPLE, which estimates a probabilistic consensus segmentation and each rater’s performance level simultaneously (Warfield et al., 2004), or fall back to per-pixel majority. Send only the genuinely discordant cases to a senior clinician.

A worked example makes the shape concrete. Say three readers screen a batch of chest CTs for a lung nodule. Two mark a nodule in the right upper lobe; one misses it. The 2-of-3 rule accepts the finding, and the case only reaches your adjudicator if you’d pre-declared that a 2-1 split is worth a look.

Now switch the task to segmenting that nodule’s volume: majority voting on a boundary is meaningless, so you’d feed the three masks to STAPLE and take the probabilistic consensus. Same data, different truth-construction rule, because the annotation type changed.

How much agreement is “enough” is its own subject. Expert annotators genuinely disagree, and that disagreement sets a ceiling on the model performance you can measure (Warfield et al., 2004). For the metrics that quantify it—and why you’d choose one over another—see Dice vs IoU vs Hausdorff for spatial overlap, and Cohen’s kappa for categorical agreement. This hub deliberately stops at “combine, then measure”; those two deep-dives carry the formulas.

How do you de-identify medical images?

De-identification means cleaning two separate places, and teams routinely forget the second. The first is the DICOM header, where names, IDs, dates, and device details live; the DICOM standard’s Basic Application Level Confidentiality Profile specifies which attributes to remove or replace (DICOM PS3.15). The second is the pixel data itself—text burned into ultrasound frames, scanned-in annotations, secondary-capture screenshots—which the header profile does not touch.

For burned-in pixel PHI, the standard defines a separate Clean Pixel Data Option that blacks out identifying text in the image (DICOM PS3.15). The catch is that locating burned-in text automatically is hard, and deciding whether a given string is identifying is harder still, so the most conservative compliant approach is to remove all burned-in text even at the cost of some useful annotation. Because automation misses cases, human inspection of each image is standard practice before an open dataset is released (Willemink et al., 2020).

How much QA do medical annotations need?

Set the review rate by clinical risk, not by a flat percentage. A quality-control flag on a screening triage model and a segmentation mask feeding a measurement that informs treatment do not deserve the same scrutiny, and spending equal QA on both wastes your experts on the low-risk half. Tie review intensity to what a wrong label would cost downstream.

Beyond agreement, computational-pathology practice measures annotations on completeness (are all required structures labeled?), exhaustiveness (is everything inside a region annotated?), diversity, and inter-annotator agreement, and runs an automatic QC pipeline that flags annotations violating the data dictionary for triage and resolution (Wahab et al., 2022). Two operational habits pay off repeatedly: a pilot phase on a handful of cases to surface protocol ambiguity before it scales, and treating inter-annotator disagreement as signal—recurring discordance usually means your codebook is unclear, not that a rater is careless (Wahab et al., 2022). Where the reader is genuinely unsure, an explicit “unknown” category beats a forced, noisy guess.

Why a curated coding scheme matters as much as the tool

The most underrated artifact in any annotation project is the coding scheme—the data dictionary that says exactly what each label means, with definitions precise enough that two annotators reach the same answer. Interoperability and reuse depend on it: an annotated dataset is only useful to the next team if it ships with the dictionary that defines its terms (Wahab et al., 2022). A great tool with a vague scheme produces confident, inconsistent labels.

Clinical measurement solved this problem decades ago, and imaging teams can borrow the pattern. A validated rating scale is a curated coding scheme with anchored definitions, a fixed structure, and a citation trail—look at how the MADRS specifies each depression item, or how the PHQ-9 fixes its nine items and scoring. That’s the bar an image-annotation data dictionary should aim for: every label defined, every ambiguity pre-resolved, the whole scheme documented and versioned. The scheme is the part that makes agreement possible; the tool just records it.

Where Tagaroo fits

Tagaroo is a schema-first annotation workspace: you define a coding scheme once, then tag against it—text, images, or samples—with inter-rater reliability and adjudication built into the workflow rather than bolted on afterward. The design bet is that most annotation projects live or die on the scheme and the agreement mechanics, which is exactly where general-purpose tools are thinnest.

Be clear about the boundary. If your job is industrial-scale pixel segmentation of DICOM volumes with specialized auto-annotation, a dedicated computer-vision platform is the right call—we compare the main ones in CVAT vs Labelbox vs V7. Tagaroo’s strength is guided, schema-driven annotation—classification and region tagging against a curated taxonomy, with every label anchored to the evidence that justifies it and agreement computed as annotators work. The same engine that codes a clinical interview against a scale also tags findings on an image against a defined scheme; if your work sits closer to transcripts, start with clinical transcript annotation tools.

On data handling, Tagaroo takes the de-identify-first path this guide argues for, rather than positioning itself as a PHI processor: its terms require you to strip direct identifiers—DICOM header fields and burned-in pixel PHI—before upload, its anonymous trial mode runs entirely in the browser so trial images never leave your machine, and the stack is EU-hosted and GDPR-oriented. It is not a medical device, and any AI pre-label is a first pass for a qualified human to review, never a diagnosis. If your project must process raw PHI in the cloud under a signed Business Associate Agreement, put that question to any vendor—including this one—before uploading identifiable studies; see Tagaroo’s privacy policy for specifics.

The practical upshot

Medical image annotation is a measurement pipeline, not a drawing exercise. Get the sequence right—collect, de-identify, annotate, adjudicate, QA, export—define your reference standard before anyone labels, match annotators to clinical risk, combine multiple reads into consensus with a pre-specified rule, and clean PHI from both the header and the pixels. Do that and the label becomes what it was always supposed to be: trustworthy ground truth.

Start with the coding scheme, because it decides how much you’ll fight later. If you want a workspace that treats the scheme and inter-rater reliability as first-class from label one, build your annotation scheme in Tagaroo—and use the cluster deep-dives above for the parts that need the math.

References

• Willemink MJ, Koszek WA, Hardell C, et al. Preparing Medical Imaging Data for Machine Learning. Radiology. 2020;295(1):4–15. doi:10.1148/radiol.2020192224
• Wahab N, Miligy IM, Dodd K, et al. Semantic annotation for computational pathology: multidisciplinary experience and best practice recommendations. Journal of Pathology: Clinical Research. 2022;8(2):116–128. doi:10.1002/cjp2.256 (preprint arXiv:2106.13689, 2021)
• Warfield SK, Zou KH, Wells WM. Simultaneous Truth and Performance Level Estimation (STAPLE): An Algorithm for the Validation of Image Segmentation. IEEE Transactions on Medical Imaging. 2004;23(7):903–921. doi:10.1109/TMI.2004.828354
• DICOM Standards Committee (NEMA). PS3.15, Annex E: Attribute Confidentiality Profiles—Basic Application Level Confidentiality Profile and Clean Pixel Data Option. dicom.nema.org
• Innolitics. Definitive Guide to AI/ML SaMD Ground Truthing: Adjudication Strategies. 2025. innolitics.com
• Encord. How to Structure QA Workflows for Medical Images. 2022. encord.com