How to Run a Pilot Annotation Round Before You Scale

A pilot annotation round is a small, deliberate dry run of your coding process—a handful of items, at least two independent coders, and a measured agreement number—that you run before you commit the full labeling budget. Its job is to surface the broken category definitions, the ambiguous boundaries, and the miscalibrated coders while they are still cheap to fix. Skip it, and you find those same problems after ten thousand items are already labeled, when the only remedy is relabeling them (Pustejovsky & Stubbs, 2012).

This is a working guide to running that pilot: how to scope it, how many items to sample, how to double-code and measure agreement, and how to turn the disagreements you find into codebook revisions instead of arguments. The examples are synthetic clinical-transcript snippets, but the mechanics transfer to any labeling task, from named-entity spans to image classes.

What a pilot annotation round is, and what it is not

A pilot annotation round tests the codebook, not the coders. You take a small representative slice of the data, have two or more people label it independently under the draft guidelines, measure how often they agree, and read the disagreements as evidence about which definitions are underspecified. The output is not a dataset you keep; it is a list of fixes to your scheme.

That framing rules out two common mistakes. The first is treating the pilot as a warm-up whose labels you fold into the final corpus, which quietly contaminates the data with output from a codebook you already know was broken. The second is treating it as a one-shot exam the team either passes or fails, when its real value is diagnostic: it tells you where interpretation diverges so you can go fix that specific boundary. A pilot is closer to a code review than to a final grade.

It also has a defined stopping condition, which most informal “let’s try a few and see” efforts lack. You pilot, revise, and re-pilot until agreement stabilizes above a threshold you chose in advance, then you scale. Everything below is how to run that loop without fooling yourself.

Why run a pilot annotation round before you scale?

You pilot because a codebook error is an upstream defect, and upstream defects get more expensive the longer they travel. Fix a vague definition during a 40-item pilot and you edit one paragraph; discover the same flaw after 8,000 items are coded and you relabel 8,000 items, because everything downstream inherited the wrong boundary (Pustejovsky & Stubbs, 2012). The pilot is the cheapest place in the whole project to buy that correction.

The failure has a name in the data-centric machine-learning literature: small, neglected problems in how data is defined and collected compound into large, hard-to-trace failures later, an effect documented as data cascades in applied AI. A pilot is the intervention that interrupts the cascade at its source, before the bad labels harden into a dataset other people trust. The same logic sits behind the effort to find label errors in your dataset after the fact: catching the error early is strictly cheaper than hunting it down later.

There is a quieter payoff, too. Agreement is not a fixed property of a task; it moves with decisions you control. In Bayerl and Paul’s (2011) meta-analysis of 96 annotation studies, trained coders averaged 81% agreement against 70% for untrained ones, and agreement fell steadily as schemes added categories. A pilot round is where that training happens and where you discover that your eleven-category scheme should have been six.

The pilot annotation round in seven steps

Run the pilot as an ordered loop. Each step has a purpose and a primary-source justification, and the list is deliberately self-contained so you can lift it straight into a protocol.

1. Scope. Fix the a priori decisions before anyone codes: which construct the pilot tests, how many coders, the unit of coding (sentence, turn, or meaningful chunk), the agreement coefficient, and the threshold that counts as acceptable (O’Connor & Joffe, 2020).
2. Sample. Draw a small, representative subsample that contains every category at least once, sized deliberately rather than by gut, since a reliability check needs each value present to be evaluable (Lacy & Riffe, 1996; Neuendorf, 2017).
3. Double-code. Have at least two coders independently label the same items with no consultation, because reliability data only mean something when coders are interchangeable and work separately on the same units (Krippendorff, 2004).
4. Measure. Compute a chance-corrected agreement coefficient for each category rather than raw percent agreement, and interpret it against the threshold you set in step 1 (Landis & Koch, 1977; Krippendorff, 2004).
5. Adjudicate. Sit with every disagreement, resolve each hard case, and record the ruling and its rationale in a dated decision log so the resolution becomes shared precedent (MacQueen et al., 1998).
6. Revise. Rewrite the specific definitions that produced low agreement, add a contrastive example for each boundary that blurred, and collapse categories the scheme did not need (Bayerl & Paul, 2011).
7. Re-pilot. Re-run the loop on a fresh subset of data until agreement stabilizes above threshold, then scale; if it is still climbing, you are not done (Pustejovsky & Stubbs, 2012; O’Connor & Joffe, 2020).

The order matters. Steps 1 through 4 are a measurement; steps 5 and 6 are the repair; step 7 is the check that the repair worked. Teams that skip straight from “we disagreed” to “let’s just discuss it” collapse adjudication and revision into a hallway conversation that never reaches the codebook, and the fix evaporates the moment the meeting ends.

How many items should a pilot annotation round include?

Size the pilot’s reliability subsample on purpose, not by picking a round percentage. A widely cited heuristic is that the reliability sample should be no fewer than 50 units or 10% of the full set, and rarely needs to exceed about 300, with larger samples required when the corpus is large or expected agreement is low (Neuendorf, 2017). That range is a floor for stability, not a target to pad toward.

The more defensible route treats reliability itself as a sampling problem. Lacy and Riffe (1996) derive the required number from four inputs: the total units to be coded, the confidence level you want, the precision you need in the agreement estimate, and an estimate of the agreement you would see if every unit were double-coded. The percentage-of-corpus rule tends to give you too few units for a small dataset and far too many for a large one; a formula avoids both. Whichever you use, reporting guidelines expect the sample size and its rationale to be stated explicitly rather than left implicit (Kottner et al., 2011).

One structural requirement overrides the arithmetic: every category has to appear in the pilot sample, or you cannot estimate its reliability at all. Rare labels are exactly the ones a pilot needs to stress-test, so if a straight random draw misses a category, sample additional items until each one is represented. This is the same question that governs how many people should label each item, which we work through in how many annotators you need for a labeling task.

How do you measure agreement in a pilot?

Measure agreement with a chance-corrected coefficient computed per category, because raw percent agreement rewards you for the easy, common labels and hides disagreement on the rare, hard ones. Two coders who both mark “no thought disorder” on 95% of turns will post a gaudy percent agreement while disagreeing on nearly every clinically interesting span. A coefficient that corrects for chance strips that illusion out.

Interpret the number against a threshold set before coding, not one reverse-engineered to make the pilot look good. Landis and Koch’s (1977) widely used bands read 0.61 to 0.80 as substantial and 0.81 upward as near-perfect, while Krippendorff (2004) recommends alpha at or above 0.800 for firm conclusions and 0.667 as the lowest floor at which even tentative conclusions are defensible. These are conventions, not laws of nature, and Krippendorff is explicit that the cutoffs should be justified by the cost of drawing a wrong conclusion, which in clinical work is high.

Which coefficient to reach for depends on your coders, categories, and measurement level. Cohen’s kappa suits two coders on nominal labels, weighted kappa handles ordered severity, and Krippendorff’s alpha generalizes to many coders and missing data; the trade-offs are laid out in our guide to choosing the right IRR coefficient and, for the two-coder case, in Cohen’s kappa and inter-rater reliability.

Turning pilot disagreements into codebook revisions

The disagreements are the point of the pilot, so mine them rather than smoothing them over. Each split between two conscientious coders marks a place where the definition let them down, and the most useful thing you own after a pilot is the list of exactly which cases they read differently. Disagreement here is signal, not noise to be scrubbed: it localizes the ambiguity for you.

Work each hard case into a ruling and record it in a dated decision log, the practice MacQueen and colleagues (1998) built into team codebook development at the CDC, where a structured, iteratively revised codebook was shown to improve intercoder agreement across dispersed coders. Every entry names the case, the question it raised, the decision, and the rule that decision produced, so the tenth coder inherits the first coder’s resolved boundary instead of re-deciding it. Then rewrite the failing definitions themselves: add the exclusion rule and the contrastive negative example that would have prevented the split, following the mechanics in annotation guidelines that reduce disagreement.

O’Connor and Joffe (2020) fold this into their suggested procedure directly: codes that fall short of the threshold are examined to identify why interpretation diverged, then removed or revised, and the revised frame is re-evaluated “preferably on a different subset of data.” That last clause is the discipline most teams miss. Re-testing on the same items you just argued about measures how well you memorized the answers, not whether the codebook now travels to data you have not seen.

The Model-Annotate-Model-Annotate loop

The pilot is not one round; it is a loop with a stopping rule. Pustejovsky and Stubbs (2012) describe annotation development as a Model-Annotate-Model-Annotate cycle: you build a model of the task (the codebook), annotate a small batch, inspect where the model and the data disagree, revise the model, and annotate again. A pilot round is one turn of that crank.

Each turn should get cheaper and more targeted. The first round tends to expose structural problems—too many categories, a missing “uncertain” option, two labels that no one can separate. Later rounds chase narrower boundary disputes. If round three surfaces problems as fundamental as round one did, the issue is usually the scheme’s design rather than the coders, and the fix is to simplify, since agreement falls as categories multiply (Bayerl & Paul, 2011).

A worked example: two rounds on a synthetic scheme

Here is a synthetic pilot on a small thought-disorder coding scheme, two coders, the same 40 turns of de-identified-style transcript, kappa computed per category. Round one uses a draft codebook; round two uses the codebook after adjudication and revision. The numbers are illustrative, but the shape—weak boundaries in round one, targeted fixes, recovery in round two—is what a healthy pilot looks like.

Category	Round 1 κ (draft codebook)	Fix applied after adjudication	Round 2 κ (revised)
Derailment	0.52	Added a priority rule and a contrastive example vs. Tangentiality	0.81
Tangentiality	0.49	Rule: code Tangentiality only if the reply engages the question before drifting	0.78
Circumstantiality	0.71	Clarified the 'returns to the point' inclusion criterion	0.84
Pressure of speech	0.44	Replaced the impression-based cue with observable triggers (rate, interruptibility)	0.79
Overall (per-category minimum)	0.44	—	0.78

Two things are worth reading off the table. Pressure of speech was the worst category in round one (κ = 0.44, borderline), and the fix was not more prose but a switch from an impression (“the subject seems pressured”) to observable triggers a tired coder can check. And the honest headline number is the per-category minimum, 0.44 then 0.78, not a flattering average that would have let the broken label through. The pilot earned its keep by telling you precisely which two definitions to rewrite.

What curated clinical scales give a pilot for free

If your scheme starts from a published clinical instrument, much of the pilot’s hardest work—operational definitions, anchored severity levels, contrastive rules for near neighbors—is already done and already piloted, in some cases across decades of use. A curated scale is, in effect, a codebook that has survived many rounds of the loop this article describes, which is why turning one into a working scheme is its own discipline, covered in going from a scale to a codebook.

The Scale for the Assessment of Thought, Language, and Communication is the clearest model for span labeling: each item ships with an operational definition and examples, and the glossary spells out how to separate the pairs that blur, exactly the boundaries a pilot would otherwise have to discover the expensive way. The Montgomery-Åsberg Depression Rating Scale shows the anchored-severity pattern, where defined levels for each item let different raters map the same interview onto the same score. Starting a pilot from an instrument like these means your first round tests calibration and edge cases rather than re-deriving definitions from scratch.

Where Tagaroo fits

The loop in this article—scope, sample, double-code, measure, adjudicate, revise, re-pilot—is the workflow Tagaroo is built around. Curated scales load as pre-written, battle-tested codebooks with inter-rater reliability computed as coders work, so a pilot round becomes a guided pass rather than a spreadsheet-and-argument exercise, and the disagreements surface as a worklist of definitions to fix. You are not starting from a blank manual; you are starting from an instrument that already reduced disagreement in the literature.

The practical upshot: run the pilot before you spend the budget, size the sample deliberately, measure agreement per category with a chance-corrected coefficient, turn every disagreement into a dated codebook revision, and re-pilot until the number stops moving. A pilot annotation round is the cheapest correction you will ever make on an annotation project, and skipping it is the most expensive habit you can keep. If you try Tagaroo on your own data, annotations stay scoped to your project and are never used to train shared models; see our privacy policy for specifics.

References

• Pustejovsky J, Stubbs A. Natural Language Annotation for Machine Learning: A Guide to Corpus-Building for Applications. O’Reilly Media; 2012. ISBN 9781449306663.
• O’Connor C, Joffe H. Intercoder Reliability in Qualitative Research: Debates and Practical Guidelines. Int J Qual Methods. 2020;19:1-13. DOI
• MacQueen KM, McLellan E, Kay K, Milstein B. Codebook Development for Team-Based Qualitative Analysis. Cult Anthropol Methods. 1998;10(2):31-36. DOI
• Krippendorff K. Reliability in Content Analysis: Some Common Misconceptions and Recommendations. Hum Commun Res. 2004;30(3):411-433. DOI
• Neuendorf KA. The Content Analysis Guidebook. 2nd ed. SAGE Publications; 2017. DOI
• Lacy S, Riffe D. Sampling Error and Selecting Intercoder Reliability Samples for Nominal Content Categories. Journal Mass Commun Q. 1996;73(4):963-973. DOI
• Kottner J, Audigé L, Brorson S, Donner A, Gajewski BJ, Hróbjartsson A, Roberts C, Shoukri M, Streiner DL. Guidelines for Reporting Reliability and Agreement Studies (GRRAS) were proposed. J Clin Epidemiol. 2011;64(1):96-106. DOI
• Bayerl PS, Paul KI. What Determines Inter-Coder Agreement in Manual Annotations? A Meta-Analytic Investigation. Comput Linguist. 2011;37(4):699-725. DOI
• Landis JR, Koch GG. The Measurement of Observer Agreement for Categorical Data. Biometrics. 1977;33(1):159-174. DOI

If your labeling problem is clinical language or structured text, Tagaroo turns curated schemes like the TLC and MADRS into guided, reliability-tracked workflows with the pilot loop built in.