Reliability & Agreement

10 articles

Inter-rater reliability is where annotation projects are won or lost. If two trained raters cannot agree on what a label means, no downstream statistic can rescue the dataset—and yet IRR is routinely treated as a reporting formality rather than a design problem. These articles cover the metrics themselves—Cohen's kappa, weighted kappa, Krippendorff's alpha, the intraclass correlation coefficient—and why raw percent agreement flatters almost every project that reports it.

The metric is only half the story. The other half is the study design behind the number: how many raters per item, how to run a pilot round, what an acceptable threshold actually is in your field, how to detect coder drift in a long study, and what to do when agreement comes back low. We work through the calculations step by step, with the paradoxes (high agreement, low kappa) treated as teaching cases rather than footnotes.

If you are writing the reliability section of a paper, auditing someone else's, or deciding whether your codebook is ready for a full campaign, start here.

Two horizontal confidence-interval ranges centered on the same point estimate—a wide uncertain interval above and a narrow confident one below—over abstract ruler tick-marks.

Reliability & Agreement

Confidence Interval for Kappa, Alpha, and ICC, Done Right

Why a bare agreement number misleads, and how to build a confidence interval for kappa, alpha, and the ICC. See analytic vs bootstrap methods.

Jul 22, 2026Read

Two columns of abstract tick-marks linked by thin green lines showing matching coder labels, a teal agreement band wrapped by a coral confidence-interval bracket, above a faint dashed chance-floor line.

Reliability & Agreement

Percent Agreement, Reconsidered: When It's Actually Fine

Raw percent agreement isn't useless. When to chance-correct, when kappa misleads, and when agreement plus a confidence interval is the honest report.

Jul 22, 2026Read

A speech waveform on a warm off-white field, segmented by two annotators into bracketed spans in green and teal; where their brackets overlap but do not align, the shared stretch glows coral, and a small tick-scale measures the gap between the two boundary marks.

Reliability & Agreement

Span-Level Agreement: Why Kappa Fails, Use F1 and IoU

Cohen's kappa needs a fixed item set that span and NER annotation never has. Measure span-level agreement with pairwise F1 and IoU instead—see how.

Jul 21, 2026Read

One column of rating nodes read by two slightly offset rulers and a fan of dial gauges whose needles point to different values, illustrating how different ICC forms give different reliability numbers from the same data.

Reliability & Agreement

Which ICC to Use: Choosing Model, Type, and Definition

Which ICC to use for continuous ratings: choose the model (1, 2, or 3), single vs average measures, and absolute agreement vs consistency, from one table.

Jul 21, 2026Read

Two identical gauges read the same heavily skewed pile of tokens: the left needle collapses toward zero while the right needle holds high, illustrating how one chance correction breaks under lopsided prevalence while another stays stable.

Reliability & Agreement

Gwet's AC1 and AC2: A Fix for Kappa's Prevalence Paradox

Why Cohen's kappa collapses under skewed prevalence, and how Gwet's AC1 and AC2 chance-correction fixes it—with the formula and a worked example.

Jul 21, 2026Read

An abstract grid of paired tick-marks where most pairs align in green and teal, a few sit slightly offset to show disagreement, one cell is left blank to represent a missing rating, and a single coral mark highlights the lone divergence.

Reliability & Agreement

Krippendorff's Alpha: Nominal, Ordinal & Missing Data

How to compute Krippendorff's alpha: the observed-over-expected disagreement formula for nominal, ordinal, and missing data, with a worked example.

Jul 21, 2026Read

Abstract diagram of a reliability study: a rising grid of subject-and-rater nodes beside a stack of confidence-interval brackets that narrow as the sample grows, with one target-width bracket highlighted in coral.

Reliability & Agreement

Sample Size for Inter-Rater Reliability: Subjects and Raters

Set the sample size for inter-rater reliability: how many subjects and raters pin kappa or an ICC to a target CI width. See the planning tables.

Jul 21, 2026Read

A methods paragraph shown as a row of bracketed blank slots being filled with green and teal blocks beside a checklist of tick-marks, with one slot marked by a coral confidence-interval bracket.

Reliability & Agreement

How to Report Inter-Rater Reliability: A Methods Template

A fill-in-the-blanks methods template for how to report inter-rater reliability: coefficient, raters, unit of analysis, CI, and software, per GRRAS.

Jul 21, 2026Read

An abstract decision-fork diagram in which five differently shaped marker groups—scattered dots, a graded staircase of ticks, a smooth curve, a bracketed span, and a small grid—flow down branching channels into separate collection bins, with one branch highlighted in coral.

Reliability & Agreement

Which Inter-Rater Reliability Coefficient Should You Use?

Which inter-rater reliability coefficient to use, by data type, rater count, and missing data—a decision guide to kappa, alpha, AC1, and ICC.

Jul 21, 2026Read

Two columns of tick-marks linked by green lines over a faint 2×2 grid, with a shaded "chance" band being subtracted to reveal the smaller chance-corrected agreement, one grid cell accented in coral—illustrating Cohen's kappa.

Reliability & Agreement

Cohen's Kappa: How to Compute and Interpret Agreement

How to compute Cohen's kappa and weighted kappa by hand, read the result honestly, and know when to use Krippendorff's alpha, ICC, or Gwet's AC1 instead.

Jul 13, 2026Read