Krippendorff's Alpha: Nominal, Ordinal & Missing Data

Krippendorff’s alpha is the one agreement coefficient that does not fall apart when a study gets realistic: more than two coders, a rating scale instead of a yes/no label, and the inevitable missing cell where someone skipped an item. Its definition is a single ratio, α = 1 − Dₒ / Dₑ, one minus observed disagreement over the disagreement you would expect by chance, and everything else is bookkeeping for how you measure “disagreement.” This guide gives the exact formula, a worked example you can reproduce by hand, the difference functions for nominal, ordinal, and interval data, and a clear rule for when to reach for alpha instead of Cohen’s kappa.

What is Krippendorff’s alpha?

Krippendorff’s alpha is a reliability coefficient that measures how much coders, raters, or measuring instruments agree beyond chance, generalized so that one formula covers any number of coders, any level of measurement, and missing data (Krippendorff, 2011). It emerged in content analysis but applies wherever two or more methods generate data on the same units and you need to know how far the resulting data can be trusted (Hayes & Krippendorff, 2007).

The coefficient is anchored at two reference points. When coders agree perfectly, observed disagreement is zero and α = 1, perfect reliability. When they agree only as much as random labeling would produce, observed disagreement equals expected disagreement and α = 0, the absence of reliability (Krippendorff, 2011). Values can go negative when disagreement is systematic rather than random.

What sets alpha apart from most agreement statistics is that it corrects for chance using the pooled distribution of all values across all coders, not each coder’s own habits. That design choice is why alpha is a genuine generalization: drop it to two coders and nominal categories and it becomes Scott’s pi, not Cohen’s kappa (Krippendorff, 2011). It is a family of coefficients wearing one name.

The formula: observed over expected disagreement

Alpha is one minus the ratio of two disagreements: what you observed, and what chance would have produced. Written out:

The pieces are all counted from a coincidence matrix, which is where alpha differs from kappa’s contingency table. A contingency table counts units; a coincidence matrix counts values, tabulating every pair of ratings that occur on the same unit (Krippendorff, 2011). The terms mean:

• o(c,k) is the observed number of times values c and k are paired within the same unit.
• n(c) is the sum of row c, the total number of c values that can be paired.
• n is the grand total, every pairable value across all units.
• δ²(c,k) is the difference function, the squared distance between values c and k, set by the measurement level.

For nominal data, where δ² is 0 when the values match and 1 when they differ, the ratio collapses into a form you can compute by hand:

A worked example: three coders and a missing value

Here is the case Cohen’s kappa cannot even accept: three coders, four units, and one skipped rating. Two coders would be fine for kappa; three plus a gap is exactly where alpha earns its place. Each unit is a synthetic utterance flagged for a phenomenon as present (1) or absent (0).

Unit	Coder A	Coder B	Coder C	Values (mᵤ)
1	1	1	1	3
2	0	0	absent	2
3	1	0	1	3
4	0	0	1	3

First, build the coincidence matrix. Within each unit, every ordered pair of values contributes, weighted by 1 / (mᵤ − 1), where mᵤ is the number of values in that unit. Unit 1 holds three 1s, so its six ordered 1-1 pairs add 6 / (3−1) = 3 to the “1,1” cell. Unit 2 holds two 0s (Coder C is absent), adding 2 / (2−1) = 2 to the “0,0” cell.

Units 3 and 4 each split, contributing one matching and two mismatching half-pairs apiece. A unit with only one value would afford no pairs and drop out on its own, which is exactly how missing data is absorbed rather than dropped.

Tallying every unit gives the symmetric coincidence matrix:

	Value 0	Value 1	Margin n(·)
Value 0	3	2	5
Value 1	2	4	6
Total	5	6	11

Now plug in. The mismatched coincidences are the off-diagonal cells, 2 + 2 = 4, so observed disagreement is Dₒ = (1/11) · 4 = 0.364. Expected disagreement uses the margins: the off-diagonal chance term is n(0)·n(1) + n(1)·n(0) = 5·6 + 6·5 = 60, giving Dₑ = (1 / (11·10)) · 60 = 0.545. Therefore:

An alpha of 0.333 sits well below Krippendorff’s 0.667 floor, so these coders would be judged unreliable on this variable, an honest verdict that percent agreement (a flattering 7 of 11 pairs match) would have hidden.

Nominal, ordinal, interval: choosing the difference function

Alpha adapts to the measurement level by swapping one component, the difference function δ²(c,k), while the rest of the formula stays fixed (Krippendorff, 2011). This is the feature that lets a single coefficient serve categorical labels and graded severity ratings without changing tools.

Level	Difference function δ²(c,k)	Typical use
Nominal	0 if c = k, otherwise 1	Speaker role; a phenomenon present vs absent
Ordinal	Squared distance in accumulated rank frequencies	Severity ratings such as the MADRS or TLC
Interval	(c − k)²	Scores on an equal-interval scale
Ratio	((c − k) / (c + k))²	Counts, durations, absolute magnitudes

Ordinal alpha is the subtle one, and the reason “ordinal alpha” is worth naming separately. It does not use the plain gap between rank numbers; it weights a disagreement by the accumulated frequencies of every rank lying between the two values, squared (Krippendorff, 2011). The practical effect matches intuition: on a severity scale, rating an item “mild” when a colleague rated it “moderate” is a small disagreement, while “none” versus “severe” is a large one, and ordinal alpha charges the second far more than the first. That is the same logic as weighted kappa, generalized to any number of coders and to missing data.

This is where alpha meets real instruments. A clinician-rated scale like the Montgomery-Åsberg Depression Rating Scale (MADRS) produces ordinal severity per item, so ordinal alpha (or an intraclass correlation on the total) is the honest reliability statistic. Andreasen’s Scale for the Assessment of Thought, Language and Communication (TLC) is the same shape: each disorder of speech is rated by graded severity, where a one-step coder disagreement should not count like a three-step one.

How is Krippendorff’s alpha different from Cohen’s kappa?

Krippendorff’s alpha lifts the three constraints baked into Cohen’s kappa: two coders only, nominal categories only, and complete data only (Cohen, 1960; Hayes & Krippendorff, 2007). Kappa is the right tool when all three hold and a reviewer expects it. The moment any one breaks, alpha is the safer default.

Property	Cohen's kappa	Krippendorff's alpha
Number of coders	Exactly 2	Any number
Measurement level	Nominal only	Nominal, ordinal, interval, ratio
Missing data	Not allowed (listwise drop)	Handled natively
Chance model	Each coder's own marginals	Pooled distribution of all values
Two-coder nominal limit	Cohen's kappa	Reduces to Scott's pi

The chance-model row is the one people miss. Cohen’s kappa estimates expected agreement from each coder’s individual rate of using each category, which lets an idiosyncratic coder inflate or deflate the correction. Alpha pools all coders into one distribution, so it does not reward two coders for having complementary biases (Krippendorff, 2011).

That is why alpha and kappa can disagree even on two-coder nominal data, and why alpha coincides with Scott’s pi rather than with kappa in that limit. For the full mechanics of the two-rater case, including the prevalence paradox that trips up kappa, see our guide to computing and interpreting Cohen’s kappa.

The same overlap-versus-category split shows up in images: a pixel mask compared with Dice, IoU, and STAPLE is an overlap problem, while a chance-corrected coefficient like alpha or kappa is for categorical labels. Pick the coefficient from the data, not from habit.

Interpreting alpha: thresholds and confidence intervals

Alpha is judged against explicit reliability standards, not vague labels. Krippendorff (2004) recommends relying on a variable only when alpha ≥ 0.800, and treating 0.667 ≤ alpha < 0.800 as adequate only for tentative conclusions; below 0.667, the data are too unreliable to support findings. These cut-offs are deliberately stricter and better justified than the popular Landis and Koch kappa bands, and Krippendorff ties the acceptable threshold to the cost of drawing a wrong conclusion from noisy data.

A point estimate is not enough, because a small sample makes alpha itself uncertain. Hayes and Krippendorff (2007) recommend reporting a bootstrapped confidence interval: resample the reliability data many times, recompute alpha each time, and report the interval that results. An alpha of 0.80 from 20 units and an alpha of 0.80 from 500 units are not the same claim, and the interval is what makes the difference visible.

Computing Krippendorff’s alpha in Python and R

You will not compute the coincidence matrix by hand in production, and you should not have to. Both languages have vetted implementations that accept a coders-by-units table with an explicit marker for missing ratings, and both reproduce the worked example above exactly (alpha = 0.333).

In Python, the krippendorff package takes a NumPy array with np.nan for missing values:

import numpy as np
import krippendorff

# rows = coders, columns = units; np.nan marks a missing rating
reliability_data = np.array([
    [1, 0,      1, 0],   # Coder A
    [1, 0,      0, 0],   # Coder B
    [1, np.nan, 1, 1],   # Coder C (skipped unit 2)
])

alpha = krippendorff.alpha(
    reliability_data=reliability_data,
    level_of_measurement="nominal",
)
print(round(alpha, 3))   # 0.333

In R, irr::kripp.alpha() expects raters in rows, subjects in columns, and NA for missing:

library(irr)

ratings <- matrix(c(
  1, 0,  1, 0,   # Coder A
  1, 0,  0, 0,   # Coder B
  1, NA, 1, 1    # Coder C
), nrow = 3, byrow = TRUE)

kripp.alpha(ratings, method = "nominal")$value   # 0.333

Switch level_of_measurement (or method) to "ordinal", "interval", or "ratio" to change the difference function, no other edits required. The irrCAC package in R adds analytic standard errors, and the original KALPHA macros for SPSS and SAS (Hayes & Krippendorff, 2007) remain a reference implementation with bootstrapping built in.

Where alpha fits in transcript annotation

For content analysis and transcript coding, Krippendorff’s alpha is usually the right first choice, because those projects routinely have more than two coders, ordinal severity scales, and missing ratings all at once (Hayes & Krippendorff, 2007). The coefficient was built for exactly this mess, which is why it became the reporting standard in the field.

In Tagaroo, each coding scheme runs as a live annotation workflow, and inter-rater reliability is computed as coders work, with the difference function matched to the scale’s measurement level. Ordinal scales get ordinal alpha; nominal span labels get the nominal form; the confidence interval tightens as the sample grows. If you care about defensible reliability, pairing the right coefficient with an auditable, reproducible scoring trail matters as much as the number itself.

The practical upshot: alpha is not a more complicated kappa, it is the general case that kappa is a corner of. Reach for it whenever you have more than two coders, a scale above nominal, or a single missing cell, and report it with an interval and the measurement level you assumed. That is the difference between a number and a defensible reliability claim.

References

• Krippendorff, K. (2011). Computing Krippendorff’s Alpha-Reliability. Departmental Papers (ASC), Annenberg School for Communication, University of Pennsylvania. repository.upenn.edu/asc_papers/43
• Hayes, A. F., & Krippendorff, K. (2007). Answering the call for a standard reliability measure for coding data. Communication Methods and Measures, 1(1), 77–89. doi.org/10.1080/19312450709336664
• Krippendorff, K. (2004). Reliability in content analysis: some common misconceptions and recommendations. Human Communication Research, 30(3), 411–433. doi.org/10.1111/j.1468-2958.2004.tb00738.x
• Krippendorff, K. (2004). Content Analysis: An Introduction to Its Methodology (2nd ed.). Sage Publications.
• Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1), 37–46. doi.org/10.1177/001316446002000104