Whole-Slide Image Annotation: A Guide for Pathology AI

Whole-slide image annotation is the process of turning a digitized pathology slide into structured ground truth a model can learn from—a label for the whole slide, a polygon around a tissue region, or a point on a single cell. It looks like drawing on a picture. In practice it’s a measurement problem at an unusual scale: the “picture” is a gigapixel, multi-resolution image, the truth is often contestable, and the person qualified to settle it is a scarce specialist.

That combination is why computational pathology projects rarely fail on the model. They fail on the annotation—on unclear label definitions, on annotating at the wrong resolution, on burning pathologist hours where a slide-level label would have done. This guide walks the parts that decide whether your dataset is worth training on.

What is whole-slide image annotation?

Whole-slide image annotation is labeling a digitized pathology slide with the ground truth a supervised model trains and is tested against. The label can be a whole-slide class (“tumor” vs “benign”), a region polygon around an anatomical structure or the invasive front, or a cell-level point or mask for tasks like mitosis counting. The annotation is only useful if it’s tied to a defensible notion of truth, which in pathology usually means a qualified reader and, where possible, an orthogonal reference such as immunohistochemistry (Niazi et al., 2019).

The word that reframes everything is scale. A whole-slide image is not one photograph; it’s a pyramid of the same tissue at many magnifications, because a pathologist reads architecture at low power and cytology at high power (Campanella et al., 2019). Annotation inherits that structure: a label is meaningful only alongside the resolution it was made at.

Why is whole-slide image annotation so hard?

The core difficulty is size. A slide digitized at 40x magnification sits around 0.25 micrometers per pixel, so a single image can exceed 100,000 by 100,000 pixels—on the order of 10 gigapixels, and tens of gigabytes uncompressed (Niazi et al., 2019). You cannot open that in an image editor, and no annotator labels it in one pass; the file is a tiled, multi-resolution pyramid that tools stream on demand.

Everything else compounds from there. Below is the short list of what actually bites teams, and the practical response to each.

Challenge	Why it bites	Practical response
Gigapixel size	A 40x slide can exceed 100,000 by 100,000 px, far too large to load or label at once (Niazi et al., 2019)	Tile the pyramid; annotate at the coarsest scale the task allows
Multi-resolution context	Architecture reads at low power, cytology at high power (Campanella et al., 2019)	Let annotators zoom freely; record the magnification each label was made at
Stain and scanner variability	Color, contrast, and focus drift across labs, stains, and scanners	Normalize and QC; flag and exclude out-of-focus regions before labeling
Ambiguous boundaries	Cell edges and tumor margins are genuine judgment calls	Adjudicate; treat recurring disagreement as a codebook signal (Wahab et al., 2022)

The multi-resolution point deserves emphasis because it’s a design decision, not a detail. Campanella and colleagues found the best magnification was task-dependent: the highest magnification worked best for prostate, while 5x was more accurate for basal cell carcinoma detection (Campanella et al., 2019). If you annotate at a fixed zoom out of habit, you may be labeling at the wrong scale for your question.

What are the levels of whole-slide image annotation?

There are three levels of whole-slide image annotation, and choosing the right one is the single highest-impact decision in the project. They trade cost against spatial precision: slide-level labels are cheap and often derivable from the pathology report, while cell-level labels are expensive and where annotators disagree most.

Level	What you label	Typical use	Cost and who
Slide-level (weak)	One label per slide, e.g. tumor / benign	Triage, weakly-supervised training, slide retrieval	Low; often derivable from the report (Campanella et al., 2019)
Region / ROI	Polygons around tissue structures or the invasive front	Tumor-bed maps, tissue-type segmentation, sampling	Medium; trained annotators working under a dictionary
Cell / nucleus	Points or masks on individual cells or nuclei	Mitosis counting, cell classification, grading	High; pathologist-anchored, the most disagreement

The practical rule mirrors the one from our medical image annotation guide: annotate at the coarsest level your clinical question allows. If a slide-level label answers it, don’t pay for cell masks. For the region and cell levels, the agreement math matters as much as the drawing—see image segmentation agreement with Dice, IoU, and STAPLE for how to quantify overlap between two annotators’ masks and combine them into consensus.

When can you use weak and slide-level labels?

Use weak labels whenever the reported diagnosis already carries the signal you need, because it’s the difference between a dataset you can build and one you can’t. The landmark demonstration is Campanella and colleagues’ multiple-instance-learning system, trained on 44,732 whole-slide images from 15,187 patients using only slide-level diagnosis labels, with no pixel-wise annotation at all (Campanella et al., 2019).

The operational payoff is what makes this worth your attention. The same study reported that such a system could let pathologists exclude 65–75% of slides while retaining 100% sensitivity on the retained set—a triage win that needs no manual outlining (Campanella et al., 2019). The catch is honest: weak supervision needs scale to work, and the axillary lymph node task (the smallest dataset) posted the weakest numbers, consistent with the finding that larger datasets are needed to drive real-world error down.

So the decision is a genuine trade. If you have volume and a report-derived label, weak supervision buys you a dataset almost for free. If you need to localize, grade, or measure, you’ll pay for region or cell annotation—and you’ll want the labor budget clear-eyed, because expert annotation time is the real cost, as we argue in the human cost of data labeling.

Why the annotation data dictionary matters more than the tool

The most underrated artifact in whole-slide image annotation is the data dictionary: the document that says exactly what each label means, with definitions precise enough that two annotators reach the same answer. A great tool with a vague dictionary produces confident, inconsistent labels—and an annotated dataset is only reusable by the next team if it ships with the dictionary that defines its terms (Wahab et al., 2022).

Best-practice pathology annotation doesn’t stop at definitions. Wahab and colleagues measure annotations on four dimensions and run an automatic quality-control pipeline that flags anything violating the dictionary for triage and resolution (Wahab et al., 2022):

Quality dimension	The question it answers
Completeness	Are all required structures labeled?
Exhaustiveness	Is everything inside a labeled region annotated?
Diversity	Do samples span sites, stains, scanners, and grades?
Inter-annotator agreement	Do independent annotators reach the same label?

Clinical measurement solved the definition problem decades ago, and pathology teams can borrow the pattern. A validated rating scale is a curated data dictionary with anchored definitions, a fixed structure, and a citation trail. Look at how the MADRS specifies each depression item with an anchored severity gradient, or how the Thought and Language Communication (TLC) scale fixes a defined set of items with explicit criteria.

That’s the bar a tissue-annotation dictionary should aim for: every label defined, every ambiguity pre-resolved before annotation starts. Designing that taxonomy is its own craft—see image labeling taxonomy design.

What does the pathologist-in-the-loop workflow look like?

The dependable workflow keeps a qualified pathologist anchoring the ground truth while software absorbs the mechanical work. AI pre-labels a slide or region; the pathologist confirms, corrects, or rejects; disagreements route to a senior reviewer; and the corrected labels feed back to improve the next pass (Niazi et al., 2019). The point is division of labor, not replacement—the model handles the gigapixels, the human owns the judgment.

The evidence for keeping the human in the loop and for what automation can do sits in the same study. In CAMELYON16, the best algorithm reached an AUC of 0.994 for detecting lymph-node metastases, beating an 11-pathologist panel whose mean AUC was 0.810 under a realistic two-hour time constraint (Ehteshami Bejnordi et al., 2017).

Read that number carefully, because the same paper contains its own caveat. The one pathologist who worked without a time limit matched the top algorithm (AUC 0.966) but needed roughly 30 hours to assess 129 slides, and the authors concluded that clinical utility “will require evaluation in a clinical setting” (Ehteshami Bejnordi et al., 2017). The lesson isn’t that machines win; it’s that unaided expert reading is slow and time-pressured reading is error-prone, which is exactly the gap a pathologist-in-the-loop annotation workflow closes.

A concrete, synthetic example makes the shape clear. Say you’re building a dataset to flag lymph-node metastases. A model pre-labels regions of interest on each slide; a trained annotator accepts the obvious positives and negatives; the equivocal 10% go to a subspecialty pathologist who adjudicates against the report and, where available, an immunohistochemical stain. You’ve spent scarce expert time only on the cases that need it, and every accepted label carries a record of who made it and at what magnification.

How many readers each case needs is a risk decision, covered in how many annotators medical images need.

What are the best practices for whole-slide image annotation?

Best practices for whole-slide image annotation come down to deciding truth before you draw, and protecting it after. The order matters because each step inherits the mistakes of the one before it:

1. Define the task and reference standard first. Decide what your label represents—and whether imaging alone is definitive or needs pathology or outcomes to confirm it—before anyone annotates (Niazi et al., 2019).
2. Pick the annotation level deliberately. Slide, region, or cell; annotate at the coarsest level your question allows (Campanella et al., 2019).
3. Write the data dictionary. Every label defined, with worked examples and the resolution it applies at (Wahab et al., 2022).
4. De-identify slides and metadata. Strip PHI from the file’s header attributes and from any text burned into the pixels or slide label, then spot-check by hand (DICOM PS3.15).
5. Annotate blind, then adjudicate. Independent reads, disagreements escalated to a senior pathologist, and recurring disagreement fed back into the dictionary (Wahab et al., 2022).
6. QC on completeness, exhaustiveness, diversity, and agreement. Run an automatic pass that flags dictionary violations for resolution (Wahab et al., 2022).

Storage and interoperability sit alongside these. Digitized slides and their labels increasingly live in DICOM as well as vendor formats, and the same de-identification discipline that applies to radiology applies to slide metadata—our DICOM annotation and data-prep guide covers header and burned-in pixel PHI in detail. If your work spans other high-resolution imagery, the same tiling-and-pyramid problems recur in satellite and microscopy data, which we cover in geospatial and scientific image annotation.

Building a WSI dataset for a regulated product

If the dataset feeds a diagnostic product, ground-truthing becomes a documentation exercise as much as an annotation one. Regulators care about who annotated, under what protocol, how disagreements were resolved, and how representative the data is—the completeness and diversity dimensions above become audit evidence, not just quality hygiene. We treat the regulatory angle separately in annotation for FDA-cleared medical AI; the short version is that a versioned data dictionary and a logged adjudication trail are the artifacts a submission leans on.

Where Tagaroo fits

Tagaroo is a schema-first annotation workspace: you define a data dictionary once, then tag against it with inter-annotator agreement and adjudication built into the workflow rather than bolted on afterward. The design bet is that most annotation projects live or die on the dictionary and the agreement mechanics, which is exactly where general-purpose tools are thinnest. For industrial-scale gigapixel segmentation with specialized slide viewers, a dedicated computational-pathology platform is the right call; Tagaroo’s strength is guided, schema-driven classification and region tagging against a curated taxonomy, with every label anchored to the evidence that justifies it.

On data handling, Tagaroo takes the de-identify-first path this guide argues for, rather than positioning itself as a PHI processor. Its terms require you to strip direct identifiers—slide and header metadata and any burned-in pixel or slide-label text—before upload; its anonymous trial mode runs entirely in the browser, so trial images never leave your machine; and the stack is EU-hosted and GDPR-oriented. Tagaroo is not a medical device, and any AI pre-label is a first pass for a qualified human to review, never a diagnosis.

We do not claim to offer a Business Associate Agreement. If your project must process raw PHI in the cloud under a signed BAA, put that question to any vendor—including this one—before uploading identifiable slides. See Tagaroo’s privacy policy for specifics.

The practical upshot

Whole-slide image annotation is a measurement pipeline, not a drawing exercise. Decide what your label represents before anyone annotates, choose the slide, region, or cell level deliberately, and remember that weak slide-level labels can carry a surprising amount of signal at scale. Write the data dictionary as if it were a rating scale, keep a pathologist in the loop to anchor the truth, and de-identify slides at both the metadata and the pixel level. Do that and the label becomes what it was always for: ground truth you can defend.

Start with the dictionary, because it decides how much you’ll fight later. If you want a workspace that treats the coding scheme and inter-annotator agreement as first-class from label one, build your annotation scheme in Tagaroo—and use the cluster deep-dives above for the parts that need the math.

References

• Wahab N, Miligy IM, Dodd K, et al. Semantic annotation for computational pathology: multidisciplinary experience and best practice recommendations. Journal of Pathology: Clinical Research. 2022;8(2):116–128. doi:10.1002/cjp2.256 (preprint arXiv:2106.13689, 2021)
• Campanella G, Hanna MG, Geneslaw L, et al. Clinical-grade computational pathology using weakly supervised deep learning on whole slide images. Nature Medicine. 2019;25(8):1301–1309. doi:10.1038/s41591-019-0508-1
• Ehteshami Bejnordi B, Veta M, van Diest PJ, et al. Diagnostic Assessment of Deep Learning Algorithms for Detection of Lymph Node Metastases in Women With Breast Cancer (CAMELYON16). JAMA. 2017;318(22):2199–2210. doi:10.1001/jama.2017.14585
• Niazi MKK, Parwani AV, Gurcan MN. Digital pathology and artificial intelligence. The Lancet Oncology. 2019;20(5):e253–e261. doi:10.1016/S1470-2045(19)30154-8
• DICOM Standards Committee (NEMA). PS3.15, Annex E: Attribute Confidentiality Profiles—Basic Application Level Confidentiality Profile and Clean Pixel Data Option. dicom.nema.org