Image Labeling Guidelines Annotators Can Actually Apply

A taxonomy fails in the field, not on the whiteboard. The image labeling guidelines that look complete in a design doc tend to fall apart the first time two annotators hit the same ambiguous photo and tag it differently. The fix is rarely more classes or a longer manual; it is definitions precise enough that a tired annotator at image 4,000 makes the same call you would. This guide is about writing image labeling guidelines annotators can actually apply: how to choose between mutually exclusive and overlapping classes, how to write a class definition that survives contact with real images, and how to capture edge cases in a decision log instead of losing them to memory.

What makes image labeling guidelines usable?

Usable image labeling guidelines are the ones where two people who never spoke to each other produce the same labels. That is the only test that matters, and it is stricter than it sounds. A manual can be exhaustive and still be unusable if the exhaustiveness lives in prose an annotator skims once and never reopens. The working definition of quality here is operational: agreement between independent annotators on held-out images, not the page count of the instructions.

This reframes the whole exercise. You are not documenting a category system; you are writing a decision procedure that a non-expert can run in a few seconds per object. Every sentence in the guidelines should change a label somewhere. If a paragraph does not resolve a choice an annotator actually faces, it is decoration, and decoration is where consistency quietly leaks away.

The stakes are higher than a noisy metric. Vague definitions are a classic data cascade: a small upstream data problem that stays invisible through training and surfaces late, at evaluation or in deployment, where it is expensive to trace and fix. In interviews with 53 AI practitioners, Sambasivan et al. (2021) found data cascades were the norm rather than the exception, and that they typically originated in underspecified data work like labeling.

Mutually exclusive vs overlapping classes: which should you choose?

Choose mutually exclusive and collectively exhaustive (MECE) classes whenever the task allows, because overlap is precisely where annotators disagree. If an object can honestly receive two labels, two conscientious annotators will split, and no amount of goodwill closes that gap. Mutually exclusive means each object belongs to exactly one class; collectively exhaustive means every object you care about has a class to go to.

The cleanest large-scale example is panoptic segmentation, which assigns every pixel exactly one class and every countable object exactly one instance id, producing a complete non-overlapping label map (Kirillov et al., 2019). That formalized the split between “stuff” (amorphous regions like road or sky) and “things” (countable objects like cars), and the value of the design is that it removes the “which label wins here?” question by construction.

Sometimes overlap is real and you cannot design it away: an object is genuinely a hybrid, or your ontology has a hierarchy where “vehicle” contains “truck.” When that happens, do not leave the tie to intuition. Make the overlap governed by rules.

Situation	Design move	Why it reduces disagreement
Classes can be made disjoint	MECE: one object, one class; add an 'other' bucket for the long tail	No object has two valid labels, so there is nothing to disagree about
True hierarchy (vehicle → truck → pickup)	Let annotators label at the level they are confident, roll up finer labels programmatically	Removes false precision; a confident coarse label beats a guessed fine one
Genuine overlap between two classes	Define a priority order: state which label wins when both apply	Turns a judgment call into a lookup
Near-neighbor classes that blur	Add contrastive rules: 'A vs B' notes with the deciding feature	Points attention at the one feature that separates them

The priority-order trick is underused and cheap. If a labeled object could be either “rider” or “pedestrian,” a single line converts a coin flip into a rule. Cityscapes does exactly this: it separates “person” from “rider” by reliance on the vehicle for movement—a rider is seated on or carried by a two-wheeler, while someone merely standing next to or pushing a bike stays “person” (Cordts et al., 2016). One sentence, and a whole family of disagreements disappears.

What does a good class definition look like?

A good class definition is a short decision procedure, not a dictionary entry. The failure mode is the one-liner: “Pedestrian: a person walking.” It reads fine and answers almost nothing, because the hard cases are the standing person, the seated person, the cyclist with one foot down, and the mannequin in a shop window. A strong definition names those cases before an annotator meets them.

The pattern that holds up has five parts: a one-line operational rule, an inclusion rule, an exclusion rule, a positive example, and a negative example. The exclusion rule and the negative example do the heavy lifting, because boundaries are defined by what is just outside the class, and prose alone rarely fixes a boundary that a near-miss image settles instantly.

Definition element	Weak version (guessable)	Strong version (applicable)
Operational rule	"Label pedestrians."	"Pedestrian = a living person primarily supported by their own feet, on or near the roadway."
Inclusion	(unstated)	Walking, standing, or sitting people; partially occluded people when the head or torso is visible.
Exclusion	(unstated)	A person in contact with a bicycle or motorcycle (label Rider); reflections, statues, mannequins, printed figures.
Positive example	(none)	A person waiting at a crosswalk, phone in hand.
Negative example	(none)	A cyclist stopped at a light with one foot down—this is Rider, not Pedestrian.
Edge-case pointer	(none)	See decision-log entries #12 (pushing a bike on foot) and #27 (child on an adult's shoulders).

Notice what the strong definition is not: it is not more verbose everywhere. It adds words only where a word changes a label. That discipline is what keeps a guidelines document usable at scale instead of becoming a wall of text nobody rereads.

Why positive and negative visual examples belong in every class

Visual examples belong in every class because the boundary between two classes is almost always easier to show than to write. A single well-chosen negative example—a near-miss that belongs to a neighboring class—resolves an ambiguity that three paragraphs of prose leave open. This is not a nicety; it is how the reference datasets were actually built.

COCO shipped exhaustive, illustrated category specifications and collected its per-instance masks as polygon outlines against those fixed definitions (Lin et al., 2014). Cityscapes published a detailed class definition document with reference imagery, and reported that annotating and quality-controlling a single image at fine pixel level took more than 1.5 hours—effort that only pays off when the class boundaries are unambiguous (Cordts et al., 2016). The lesson is consistent: reference imagery is part of the definition, not a supplement to it.

ADE20K makes the point from the opposite direction. Rather than crowdsourcing, it was labeled by a single expert annotator working against one fixed scheme, which produced markedly more consistent and denser labels—about 29 segments per image versus roughly 16 from crowd workers (Zhou et al., 2019). You will rarely have the luxury of one annotator for a whole dataset, but the takeaway transfers: consistency comes from one shared, well-specified scheme with worked examples, applied the same way by everyone.

How do you handle edge cases? Keep a decision log

Handle edge cases with a running decision log: a dated table where every hard call becomes a ruling with a short rationale and, when warranted, a new rule folded back into the guidelines. The log is the mechanism that turns a one-off judgment into shared precedent, so the tenth annotator inherits the first annotator’s hard-won calls instead of re-litigating them. Without it, a scheme drifts—each new person quietly reinvents the boundaries, and agreement decays as the team grows.

Each entry answers four questions: what was the case, what question did it raise, what did we decide, and what rule did that produce? Keep it lightweight enough that logging a case is faster than debating it in a meeting.

#	The case (synthetic)	Question	Ruling	Rule it produced
12	A person walking beside a bicycle, hands on the handlebars, not riding	Pedestrian or Rider?	Pedestrian	"Rider requires the person to be supported by the vehicle (seated/astride). Pushing a bike on foot is Pedestrian."
27	A child sitting on an adult's shoulders	One person or two?	Two Pedestrian instances	"Label each distinct person separately, even when carried."
31	A bicycle mounted on a car's rear rack	Label the bicycle?	No	"A bike carried as cargo is part of the Vehicle; label Bicycle only when free-standing or in use."
44	A mannequin in a shop window	Pedestrian?	Do not label	"Only living people are Pedestrians; exclude mannequins, statues, and printed figures."

The decision log is also the cheapest insurance you have against data cascades. Because it timestamps every ruling, you can tell exactly when a boundary changed, re-audit only the images labeled under the old rule, and avoid the situation Sambasivan et al. (2021) describe, where a labeling ambiguity is discovered far downstream with no way to trace which examples it touched.

The “when in doubt” rules that drive agreement

Every taxonomy needs a small set of default rules for the moments a specific definition does not cover, because those moments are inevitable and you would rather they resolve consistently. Three rules carry most of the weight.

• Log, don’t guess. If you cannot decide in a few seconds, apply the escape-hatch label and log the case rather than committing a silent guess. A guess looks like data and corrupts quietly; a logged uncertainty is visible and fixable.
• Prefer the confident coarse label. When a fine-grained class is uncertain but a parent class is obvious, label the parent. A correct “vehicle” beats a wrong “pickup truck,” and hierarchies let you roll finer labels up rather than fabricate precision.
• Follow the priority order. When two classes both apply, the class listed higher in the documented order wins. This is a lookup, not a judgment, and it makes ties reproducible across the whole team.

The escape hatch itself is non-negotiable, and the benchmarks bake it in. Cityscapes uses a “void” label for regions that are ambiguous or do not fit the scheme and excludes them from scoring; COCO marks dense crowds with an iscrowd flag rather than forcing individual masks (Cordts et al., 2016; Lin et al., 2014). A scheme with no way to say “unclear” forces annotators to invent one, and they will each invent a different one.

What curated clinical schemes teach image taxonomies

The discipline of overlapping-class handling is most mature in clinical language coding, and it transfers directly to images. Clinicians have spent decades writing schemes where near-neighbor categories must be separated reliably by different raters, and their answer is exactly the one above: operational definitions, verbatim examples, and explicit contrastive rules.

The clearest model is the TLC scale for thought, language, and communication, whose glossary gives each item an operational definition plus worked examples, and adds contrastive guidance for the pairs that blur—how to choose tangentiality versus derailment, or circumstantiality versus loss of goal (Andreasen, 1986). That is a decision log and a set of “A vs B” rules, formalized into a published instrument. An image taxonomy that separates “rider vs pedestrian” or “truck vs van” is solving the same problem with the same tools.

Granularity is the other shared lever. A compact scheme like Ekman’s six basic emotions buys speed and high agreement with few categories, while a finer scheme like Plutchik’s eight-emotion wheel captures more nuance at the cost of more rater disagreement (a tradeoff we unpack in image annotation types explained). The parallel to coarse boxes versus dense masks is exact: more resolution, more cost, more room to disagree. Pick the least granularity your question can tolerate.

How do you test image labeling guidelines before scaling?

Test image labeling guidelines the same way you would test any instrument: on a small pilot, with more than one annotator, measured with a chance-corrected statistic before you commit the budget. Have at least two annotators independently label the same modest set of images, compute inter-rater agreement, then treat every disagreement as a bug report against a class definition rather than a fault of the annotator.

The right metric is a chance-corrected one such as Cohen’s kappa, not raw percent agreement, because raw agreement flatters rare classes and hides the disagreements that matter (see our guide to Cohen’s kappa and inter-rater reliability). Watch which classes drive the disagreement, rewrite those definitions, add the resolved cases to the decision log, and re-run. When agreement stops climbing, the taxonomy is stable enough to scale.

Two practical notes. First, decide how many annotators you actually need for the reliability you want before the pilot, not after—a question we cover in how many annotators you need for medical images. Second, if you plan to accelerate labeling with a model, remember that a model trained or prompted on an ambiguous scheme will reproduce the ambiguity at scale; get the taxonomy stable first, then bring in model-assisted labeling to speed up the drawing, not to define the classes.

Where Tagaroo fits: taxonomy as the shared discipline

Tagaroo does not draw boxes or masks—for pixels, use a computer-vision tool, and our medical image annotation guide covers that stack. Where Tagaroo is useful is the part every modality shares: turning a coding scheme into a guided workflow and measuring whether annotators actually agree. That is the same discipline this whole article describes, applied today to structured text and clinical language.

The curated clinical scales in Tagaroo—the TLC, Ekman, and Plutchik schemes above—are worked examples of well-designed taxonomies: operational definitions, examples, and contrastive rules, with inter-rater reliability computed as annotators work. A curated library of image schemes would carry the same properties, which is the direction this points. If you try it on your own data, annotations stay scoped to your project and are never used to train shared models; see our privacy policy for the details.

The practical upshot: judge your image labeling guidelines by whether strangers agree, make classes mutually exclusive with priority rules where they cannot be, put a positive and a negative example on every class, log every edge case, and prove the scheme on a small pilot before you spend the labeling budget. Get the taxonomy right and the model work gets easier; get it wrong and you pay for it downstream, where it is hardest to fix.

References

• Sambasivan N, Kapania S, Highfill H, Akrong D, Paritosh P, Aroyo LM. “Everyone wants to do the model work, not the data work”: Data Cascades in High-Stakes AI. CHI 2021. DOI
• Lin TY, Maire M, Belongie S, Hays J, Perona P, Ramanan D, Dollár P, Zitnick CL. Microsoft COCO: Common Objects in Context. ECCV 2014. arXiv:1405.0312 · DOI
• Cordts M, Omran M, Ramos S, Rehfeld T, Enzweiler M, Benenson R, Franke U, Roth S, Schiele B. The Cityscapes Dataset for Semantic Urban Scene Understanding. CVPR 2016. arXiv:1604.01685 · DOI
• Zhou B, Zhao H, Puig X, Xiao T, Fidler S, Barriuso A, Torralba A. Semantic Understanding of Scenes Through the ADE20K Dataset. IJCV 2019. arXiv:1608.05442 · DOI
• Kirillov A, He K, Girshick R, Rother C, Dollár P. Panoptic Segmentation. CVPR 2019. arXiv:1801.00868 · DOI
• Andreasen NC. The Scale for the Assessment of Thought, Language, and Communication (TLC). Schizophr Bull. 1986;12(3):473-482. DOI

If your annotation problem is structured text or clinical language rather than pixels, Tagaroo turns curated schemes like the TLC and Ekman’s basic emotions into guided, reliability-tracked workflows.