Inter-rater reliability calculator

Inter-rater reliability answers one question: if a second qualified person coded the same material, would they reach the same conclusions? Every coefficient on this page is an attempt to answer it while correcting for the agreement you would get from two people who were guessing. They differ in how they model that guessing, which is why they can disagree sharply on the same data — and why the choice of coefficient has to be a decision you can justify rather than whichever one your software printed.

Percent agreement: honest, and not enough on its own

Percent agreement is the proportion of items two coders classified identically. It is transparent, it is the only figure most readers can interpret without training, and it should almost always be reported. Its flaw is that it credits agreement that costs nothing. If two coders each label 95% of items "absent" without looking, they will agree about 90% of the time. That is why chance-corrected coefficients exist.

Percent agreement is not obsolete, though, and the case for reporting it alongside a corrected coefficient is strong: when the corrected figure is low, percent agreement tells the reader whether that reflects genuine coding conflict or a statistical artefact of skewed categories.

Cohen's kappa: the default for two raters

Cohen's kappa (1960) subtracts the agreement expected if each rater applied their own observed base rates independently. It runs from 1 (perfect) through 0 (chance-level) to negative values (systematically worse than chance). It is the conventional choice for two raters and unordered categories, and it is what most reviewers expect to see.

The formula is simply the observed agreement, corrected for chance:

κ = (pₒ − pₑ) / (1 − pₑ), where pₒ is the observed agreement — the proportion of items the two raters classified identically — and pₑ is the agreement expected by chance, computed by multiplying the two raters' marginal rates for each category and summing across categories. The denominator is the room left above chance, so kappa reports how much of that available headroom the raters actually captured. The tool above shows pₒ and pₑ as separate readouts, which is usually more informative than the kappa alone.

Two properties surprise people. First, kappa is sensitive to how the raters distribute their codes, not only to how often they agree — two raters who use a category at very different rates are penalised. Second, and more consequentially, kappa degrades badly when one category dominates.

The prevalence paradox: high agreement, low kappa

Feinstein and Cicchetti described this in 1990, and it is the single most common source of confusion in reliability reporting. When almost every item belongs to one category, expected chance agreement is already very high, so there is little room left above it and kappa collapses — even at 85% or 90% observed agreement. The tool above flags this automatically and includes a preset that demonstrates it.

The paradox is not a bug in your data. It is a property of the statistic under skewed prevalence, and it means kappa is the wrong summary for rare phenomena — which describes a great deal of clinical coding, where the symptom of interest may appear in a handful of interviews. The fix is to report a coefficient that does not have the problem, and to say why.

Gwet's AC1: built for skewed categories

Gwet's AC1 (2008) replaces kappa's chance model with one that does not assume raters randomly guess at the observed base rates. The practical consequence is that AC1 stays stable where kappa collapses. On the paradox preset above, kappa reads about 0.13 while AC1 reads about 0.82 — describing the same 85% agreement. Neither number is wrong; they are answering slightly different questions, and AC1's is usually the one you meant to ask when one category is rare.

Scott's pi and Fleiss' kappa

Scott's pi (1955) computes chance agreement from the pooled distribution of both raters rather than each rater's own marginals, so it does not penalise raters for differing in how often they use a category. Fleiss' kappa (1971) generalises that pooled-marginal approach to any number of raters, and it does not require the same raters to code every item — only the same number of raters per item. That constraint matters in practice: if some items were coded by three people and others by two, Fleiss' kappa silently applies to a subset. The tool above reports how many items it used.

Krippendorff's alpha: the general case

Krippendorff's alpha handles any number of raters, any number of categories, and — uniquely among the coefficients here — missing data, without discarding items. It also accepts a difference metric, so it can respect measurement level: nominal treats every disagreement as total, ordinal counts adjacent categories as near-misses, and interval uses squared numeric distance. For a coding study where annotators skipped items, alpha is usually the right answer.

The implementation here reproduces Krippendorff's own published worked example to three decimal places for all three metrics (0.743 nominal, 0.815 ordinal, 0.849 interval), which is the check we would want from any tool we relied on.

Weighted kappa, for ordered categories

If your categories are ordered — absent, mild, moderate, severe — treating a mild/moderate disagreement as identical to an absent/severe disagreement throws away information. Weighted kappa (Cohen, 1968) penalises disagreements in proportion to their distance: linear weighting scales with the gap, quadratic weighting penalises large gaps much more heavily and near-misses much less. Quadratic weighted kappa is the convention for ordinal rating scales, and it is numerically equivalent to an intraclass correlation under common conditions.

For severity ratings drawn from an instrument, this is usually the coefficient you want. Our guide to choosing an IRR coefficient works through the decision in more detail, and span-level agreement covers the separate case where coders also have to agree on where in the text something happens — for which kappa is not defined at all.

Confidence intervals are not optional

A kappa of 0.62 from 30 items and a kappa of 0.62 from 300 items are not the same finding, and reporting the point estimate alone hides the difference. For Cohen's kappa this tool reports an analytic 95% interval from the Fleiss, Cohen and Everitt (1969) standard error, which is the estimator intended for interval construction. For Krippendorff's alpha it reports a seeded percentile bootstrap over items, which is what Krippendorff himself recommends; the fixed seed means a value you quote in a paper stays reproducible.

Two caveats we would rather state than bury. The interval shown for Fleiss' kappa uses Fleiss's (1971) asymptotic variance, which is derived under the null hypothesis of chance agreement — it is a serviceable approximation for a reported interval but not strictly the right quantity, and it will tend to be optimistic. And no interval at all is offered for Gwet's AC1 or weighted kappa, because we have not validated an analytic standard error for them against a published figure, and would rather show nothing than a number we cannot defend. For rigorous intervals on those coefficients, Gwet's own irrCAC package (R and Python) implements his variance estimators directly.

What counts as good agreement

The bands almost everyone cites come from Landis and Koch (1977): slight below 0.20, fair to 0.40, moderate to 0.60, substantial to 0.80, almost perfect above that. The authors themselves called the divisions arbitrary. Treat them as vocabulary, not as a threshold test.

What actually determines an acceptable value is the use. Codes feeding a primary outcome in a trial need to be more reliable than codes used to organise exploratory themes. Many health research journals expect 0.60 or better for a published coding scheme and 0.80 where the codes drive conclusions. Report the number, the interval, the coefficient, and the number of items — and let the reader judge.

If agreement is low, the codebook is usually the problem

Low reliability is rarely fixed by retraining coders on an unchanged codebook. In our experience the recurring causes are a category that bundles two distinct phenomena, a boundary defined by an adjective rather than an observable criterion, and no rule for the genuinely ambiguous case. Our guides to writing annotation guidelines and running a pilot round cover the diagnostic process; the point of a pilot is to discover these problems while the codebook can still change.

It is also worth remembering that disagreement is data. Two experienced clinicians disagreeing consistently about one category is evidence about the category, not just about the coders — a point we develop in disagreement is signal, not noise.

How to report it

The tool writes a methods sentence for you, because incomplete reliability reporting is one of the most common review comments on qualitative and observational papers. A complete report names the coefficient and its variant, the number of items and raters, the point estimate, the interval, and the software. Our IRR reporting template gives the full structure, and sample size for reliability studies explains how to size the substudy in advance — you can plan that with the sample-size planner.

Privacy

Everything on this page is computed in your browser. Codings you paste are never uploaded, and there is no account. If you are working with clinical transcripts rather than codes, the transcript de-identifier works the same way.

Frequently asked questions