inter rater reliability
Percent Agreement, Reconsidered: When It's Actually Fine
Raw percent agreement isn't useless. When to chance-correct, when kappa misleads, and when agreement plus a confidence interval is the honest report.
Two coders label 100 interview utterances for a speech feature. They agree on 86 of them. Report that as 86% agreement and a reviewer may frown at the missing chance correction.
Compute Cohen’s kappa on the identical table and it comes back at 0.05, “slight” agreement, barely above chance. Same coders, same data, two verdicts that could not disagree more.
The reflex in most methods sections is to trust the kappa and quietly bury the percent agreement. That reflex is worth questioning. Raw percent agreement is the most criticized index in the field, and also, in at least one careful controlled experiment, the one that tracked true reliability best (Zhao et al., 2022). The honest position is neither “always chance-correct” nor “raw agreement is fine”; it is knowing which situation you are in.
This piece makes the case for and against raw agreement: what it actually measures, when chance correction earns its keep, when kappa misleads through the prevalence paradox, and when raw agreement paired with a good confidence interval is the right thing to report.
What is percent agreement, and why is it criticized?
Percent agreement is the proportion of coded items on which two raters assign the same label, computed as the number of matches divided by the total. It is the oldest and simplest index of interrater reliability, and it needs no distributional assumptions: count the diagonal of the agreement table, divide by the total, done (Cohen, 1960).
The standard objection is that it counts agreement the coders would have reached by luck. If a feature is present 90% of the time, two coders who guessed “present” at random would still match about 82% of the time, so a reported 84% looks impressive and means almost nothing. Cohen (1960) built kappa precisely to subtract that chance floor, and Krippendorff (2018) makes the same argument the backbone of his case for alpha: an agreement index that does not model chance is not measuring reliability, only coincidence.
That critique is correct as far as it goes. Where it overreaches is in the leap from “percent agreement is incomplete” to “percent agreement is useless.” The chance-corrected coefficients solve the chance problem by introducing a chance model, and that model has failure modes of its own.
When is raw percent agreement actually fine?
Raw percent agreement is fine, even preferable, when the chance correction would do more harm than good and when you report the number with enough context to interpret it. Three situations qualify.
The first is many categories. As the number of label options grows, the probability of matching by chance falls fast, so the gap between raw and chance-corrected agreement shrinks toward nothing. On a 12-code scheme with roughly even use, chance agreement is already low, and percent agreement is close to what any correction would return anyway.
The second is extreme or unknown base rates. This is the counterintuitive case: exactly where kappa is supposed to help, it can break, because a lopsided base rate inflates kappa’s chance term until the coefficient collapses (Feinstein & Cicchetti, 1990). When you cannot trust the correction, the raw number plus the base rates is the more honest object.
The third is transparency. Percent agreement is the one index a clinician, a regulator, or a first-year student reads correctly without a tutorial. When the audience needs to actually understand the claim, “the coders agreed on 86% of items, 95% CI 78% to 91%” communicates more than “kappa was 0.05.” Report it beside a coefficient, never as a lone number, and it does real work. For the full menu of coefficients and when each fits, see our guide to which inter-rater reliability coefficient to use.
When does chance correction actually matter?
Chance correction matters most when the categories are few and the base rates are near-balanced, because that is the regime where a large fraction of raw agreement genuinely is chance. With two equally likely categories, two random coders match about 50% of the time, so a raw agreement of 70% is only 20 points above the coin-flip floor, and reporting the raw number alone would badly overstate reliability (Cohen, 1960).
Correction also matters when you need to compare agreement across studies or datasets with different base rates. Raw agreement is not comparable across tasks with different category distributions; a chance-corrected coefficient at least attempts to put them on a common scale. And it matters when your field simply expects it: many journals treat a kappa, pi, or alpha as table stakes, and omitting it invites a desk rejection regardless of the statistics.
The prevalence paradox: when kappa misleads
The prevalence paradox is the unsettling pairing of high observed agreement with a low or near-zero chance-corrected kappa on the very same data, and it is a property of the chance model, not of the coders (Feinstein & Cicchetti, 1990). It ambushes rare-event coding, which is most transcript annotation, because rare features produce exactly the skewed marginals that trigger it.
Here it is in one synthetic 2x2. Two coders judge 100 utterances for whether a speech feature, say circumstantiality, is present. It turns out to be present about 92% of the time.
| Coder B: present | Coder B: absent | Row total | |
|---|---|---|---|
| Coder A: present | 85 | 7 | 92 |
| Coder A: absent | 7 | 1 | 8 |
| Column total | 92 | 8 | 100 |
Observed agreement is the diagonal, Pₒ = (85 + 1) / 100 = 0.86. But both coders mark the feature present 92 times, so Cohen’s chance term climbs to Pₑ = 0.92·0.92 + 0.08·0.08 = 0.8528, leaving κ = (0.86 − 0.8528) / (1 − 0.8528) = 0.05. By the Landis and Koch (1977) bands that is “slight” agreement, one notch above nothing, on coders who matched 86 times out of 100.
Neither number lies. Kappa is reporting, correctly under its own assumptions, that when a feature is present 92% of the time, most of an 86% match is what two coders would hit blindfolded. The mistake is treating the low kappa as a coder verdict and retraining people over a base-rate artifact. This is the same machinery that motivates Gwet’s AC1, a coefficient whose chance term shrinks rather than grows under skew; we work it through in the Gwet’s AC1 and the kappa paradox deep dive.
Report positive and negative agreement, not just the total
A single percent-agreement number hides an asymmetry that the same table makes obvious once you split it. Uebersax’s raw-agreement framework and Cicchetti and Feinstein (1990) both recommend reporting specific agreement on each category: the positive agreement on “present” calls and the negative agreement on “absent” calls, computed separately.
On the table above, positive agreement is 2·85 / (92 + 92) = 0.92 and negative agreement is 2·1 / (8 + 8) = 0.125. The coders are excellent at flagging the common “present” case and nearly worthless at agreeing on the rare “absent” one. That is the real story, and it is invisible in both the 86% headline and the 0.05 kappa. Specific agreement is often the single most informative thing you can add to a reliability report on skewed data.
The evidence that percent agreement tracks true reliability
The sharpest argument for percent agreement is empirical. Zhao, Feng, Ao, and Liu (2022) ran a controlled experiment that generated ratings with known true reliabilities and chance agreements, then tested seven of the best-known indices against that ground truth across varied category counts, distribution skews, and task difficulties.
The result inverts the conventional ranking. In their words, “the most criticized index, percent agreement (aₒ), showed as the most accurate predictor of reliability” (Zhao et al., 2022). It was the only index to clear their primary benchmark, and the three most respected and most required coefficients finished last.
| Index | Predicts true reliability (dr²) | Verdict in the study |
|---|---|---|
| Percent agreement (aₒ) | 0.84 | Best predictor; only index past the 0.8 benchmark; overestimated by ~13 pts |
| Gwet's AC1 | 0.72 | Second-best predictor; most accurate approximator |
| Bennett et al.'s S | 0.69 | Third |
| Scott's π / Cohen's κ / Krippendorff's α | 0.31 | Ranked last; underestimated true reliability by 31 to 32 points |
Two cautions keep this honest. Percent agreement won as a predictor while still overestimating true reliability by about 13 points on average, so the authors suggest treating a discounted value as a crude estimate rather than reading it at face value (Zhao et al., 2022). And this is one experiment with a specific theory of how raters produce random ratings; the authors explicitly call for replication before anyone rewrites the textbooks.
It does not prove percent agreement is best. It does demolish the claim that it is worthless.
Raw agreement plus a confidence interval: the honest report
The report that survives scrutiny is raw agreement with an interval, the base rates, and a chance-corrected coefficient chosen for those base rates. Percent agreement is a proportion, so it deserves a confidence interval for a proportion; the Wilson score interval is a well-behaved default that stays within 0 to 1 and holds up at small samples (Wilson, 1927).
The 86% agreement above, on 100 items, carries a 95% Wilson interval of roughly 78% to 91%. That width is the point: a single “86%” implies a precision the data does not have, and the interval tells a reader how much of the number is sampling noise. We walk through the interval math, including for kappa and alpha, in confidence intervals for agreement.
This is not a fringe recommendation. Feng (2014), reviewing 30 years of intercoder reliability practice, argues that the deeper problem is not which index researchers pick but how thoughtlessly they report it, a single number, wrong measurement level, no interval. The fix is the same regardless of your favored coefficient: show the reader enough to interpret the number. Our copy-paste IRR reporting template lays out exactly which quantities to include.
A worked case: clinician-rated severity coding
Consider two raters applying the Montgomery-Åsberg Depression Rating Scale (MADRS) to interview transcripts, marking whether each item’s evidence is present in a passage. Most items are absent in any given interview, so the present/absent labels are skewed, precisely the regime where a lone kappa will read low even when the raters largely agree.
The right report here is not a single coefficient. It is percent agreement with a Wilson interval, the base rate of “present” for each item, positive and negative specific agreement so the asymmetry shows, and a paradox-resistant coefficient such as Gwet’s AC1 or a weighted statistic for the ordinal severity ratings. The same logic applies to span-level coding on the Scale for the Assessment of Thought, Language and Communication (TLC), where disorganized-speech features are rare by nature and skew is the default, not the exception.
So, is percent agreement useless?
No, and the question is the wrong one. Percent agreement is an incomplete number that becomes a good one the moment you surround it with context: a confidence interval, the base rates, specific agreement, and a chance-corrected coefficient matched to those base rates. Feng (2014) puts the discipline plainly, report several indices, understand each, and stop pretending a single statistic settles reliability.
The instinct to distrust raw agreement is healthy; the instinct to hide it is not. Under balanced categories, lead with a chance-corrected coefficient and note the raw agreement. Under skew, lead with raw agreement, its interval, and the base rates, and let a paradox-resistant coefficient sit beside them, because that is where kappa alone will mislead.
In Tagaroo, agreement is computed as coders work, with the coefficient matched to each scale’s measurement level and the raw agreement, intervals, and base rates surfaced together rather than collapsed into one headline. Pick the report for the base rate you actually have, not the coefficient your reflexes reach for: that is the difference between a number and an interpretation.
References
- Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1), 37–46. doi.org/10.1177/001316446002000104
- Wilson, E. B. (1927). Probable inference, the law of succession, and statistical inference. Journal of the American Statistical Association, 22(158), 209–212. doi.org/10.1080/01621459.1927.10502953
- Landis, J. R., & Koch, G. G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33(1), 159–174. doi.org/10.2307/2529310
- Feinstein, A. R., & Cicchetti, D. V. (1990). High agreement but low kappa: I. The problems of two paradoxes. Journal of Clinical Epidemiology, 43(6), 543–549. doi.org/10.1016/0895-4356(90)90158-L
- Cicchetti, D. V., & Feinstein, A. R. (1990). High agreement but low kappa: II. Resolving the paradoxes. Journal of Clinical Epidemiology, 43(6), 551–558. doi.org/10.1016/0895-4356(90)90159-M
- Gwet, K. L. (2008). Computing inter-rater reliability and its variance in the presence of high agreement. British Journal of Mathematical and Statistical Psychology, 61(1), 29–48. doi.org/10.1348/000711006X126600
- Feng, G. C. (2014). Intercoder reliability indices: disuse, misuse, and abuse. Quality & Quantity, 48(3), 1803–1815. doi.org/10.1007/s11135-013-9956-8
- Krippendorff, K. (2018). Content Analysis: An Introduction to Its Methodology (4th ed.). SAGE Publications.
- Uebersax, J. S. (2015). Raw Agreement Indices. Statistical methods for rater agreement. john-uebersax.com/stat/raw.htm
- Zhao, X., Feng, G. C., Ao, S. H., & Liu, P. L. (2022). Interrater reliability estimators tested against true interrater reliabilities. BMC Medical Research Methodology, 22, 232. doi.org/10.1186/s12874-022-01707-5
Frequently asked questions
- Is percent agreement useless?
- No. Percent agreement is incomplete on its own, not useless. In a controlled experiment that tested seven agreement indices against known true reliabilities, percent agreement was the single most accurate predictor of true interrater reliability, the only index to clear the study's primary benchmark (Zhao et al., 2022). Its real flaw is that it ignores chance agreement, so it reads high when categories are lopsided (Cohen, 1960). Report it with the base rates and a confidence interval and it becomes a usable, honest number rather than a naive one.
- When should you report percent agreement instead of Cohen's kappa?
- Report percent agreement alongside, not instead of, a chance-corrected coefficient whenever base rates are extreme, because that is exactly where Cohen's kappa produces the prevalence paradox of high agreement but a near-zero kappa (Feinstein & Cicchetti, 1990). On skewed data, raw agreement plus each category's base rate and a confidence interval describes the coders more honestly than kappa alone. On balanced categories with few options, lead with kappa, since a large share of raw agreement there is just chance (Cohen, 1960).
- Why does high percent agreement sometimes give a low kappa?
- This is the prevalence paradox described by Feinstein and Cicchetti (1990). When one category dominates, two coders agree constantly on the easy majority calls, so kappa's chance term climbs almost to the observed agreement and the corrected coefficient collapses toward zero. A synthetic table with 86% agreement can yield a kappa near 0.05 purely because the feature is present 92% of the time. The low kappa is a signal about the base rate, not proof that the coders disagree.
- What confidence interval should you use for percent agreement?
- Percent agreement is a proportion, so pair it with a confidence interval for a proportion; the Wilson score interval is a well-behaved default that stays inside 0 to 1 and works at small samples (Wilson, 1927). An agreement of 86% on 100 items carries a 95% Wilson interval of roughly 78% to 91%, which tells a reader how much sampling noise the single number hides. Reporting the interval is part of what turns raw agreement from a naive statistic into a defensible one.
- Is percent agreement the same as Cohen's kappa?
- No. Percent agreement is the raw proportion of items on which two coders assign the same label. Cohen's kappa takes that same observed agreement and subtracts the agreement the coders would reach by chance, given their marginal rates, then rescales it (Cohen, 1960). On balanced data the two move together; under skewed base rates they can point in opposite directions, which is why methodologists recommend reading them side by side (Feng, 2014).
Put this into practice
Tagaroo turns any rating scale or coding scheme into a guided annotation workflow — with inter-rater reliability computed as you go.