Gwet's AC1 and AC2: A Fix for Kappa's Prevalence Paradox

Two coders label 100 utterances for a feature that is present about 90% of the time. They agree on 84 of them. Cohen’s kappa comes back at 0.11—“slight” agreement, barely above chance—while Gwet’s AC1, computed on the very same table, reads 0.80.

Nothing about the coders changed between those two numbers; only the arithmetic of the chance correction did. That gap is the kappa paradox, and Gwet’s AC1 is the coefficient built to close it (Feinstein & Cicchetti, 1990; Gwet, 2008).

This is the deep-dive version of that story: the two paradoxes Feinstein and Cicchetti named, the exact reason kappa’s chance term detonates under skew, the AC1 and AC2 formulas that survive it, a worked example you can reproduce by hand, and an honest account of where AC1 helps and where it doesn’t.

What is the kappa paradox?

The kappa paradox is the unsettling combination of high percent agreement and a low—sometimes near-zero—chance-corrected kappa on the very same data. Feinstein and Cicchetti (1990) documented two versions of it, and both trace to how kappa estimates the agreement two raters would reach by luck alone, not to any failing of the raters.

The first version is the prevalence paradox: when one category is far more common than the others, raters agree constantly on the easy, lopsided calls, kappa attributes almost all of that agreement to chance, and the coefficient sinks toward zero. The second is the bias paradox, where the two raters’ marginal totals differ and kappa is pushed the other way—inflated for the same observed agreement. The prevalence version is the one that ambushes most annotation projects, because rare-event coding is skewed by nature.

Paradox	Cause	Symptom	What helps
Prevalence	One category dominates (base rate far from 50%)	High agreement, low or near-zero kappa	Gwet's AC1 / AC2; report the base rates (Gwet, 2008)
Bias	Raters' marginal totals differ (one says 'present' more)	Kappa inflated for the same observed agreement	Report each rater's marginals and positive/negative agreement (Cicchetti & Feinstein, 1990)

Why does Cohen’s kappa collapse under skewed prevalence?

Cohen’s kappa collapses under skew because its chance term is estimated from each rater’s marginal rates, and when one category is rare those marginals drive the expected-agreement term up toward the observed agreement, leaving almost no room above chance for the coefficient to credit. The form makes this visible: κ = (Pₒ − Pₑ) / (1 − Pₑ), observed agreement minus chance agreement, scaled by the agreement still available above chance.

When a label is present 90% of the time, both raters mark it present roughly 90% of the time out of sheer base rate, so Cohen’s chance term climbs to Pₑ = 0.90·0.90 + 0.10·0.10 = 0.82. The numerator Pₒ − Pₑ is now whatever observed agreement exceeds that 0.82, which is very little. Kappa is not measuring the coders badly; it is reporting, correctly under its own model, that most of their agreement is what two raters would hit blindfolded.

What is Gwet’s AC1?

Gwet’s AC1 is a chance-corrected agreement coefficient that keeps kappa’s overall shape but swaps in a chance term designed not to inflate when a category is rare (Gwet, 2008). It answers the same question kappa does—how much do raters agree beyond luck?—while refusing to treat lopsided base rates as if they made agreement free.

The engine is that chance term. Cohen builds expected agreement from the product of the raters’ marginals, so as a category’s rate approaches 1 the term approaches 1. Gwet builds it from q·(1 − q), the variance of a coin with bias q, which is largest at q = 0.5 and falls toward zero as q heads to either extreme. Under the same skew that maxes out kappa’s chance term, AC1’s chance term goes quiet.

Set side by side, the two coefficients differ in exactly one component, and that component is the whole story.

Property	Cohen's kappa	Gwet's AC1
Overall form	(Pₒ − Pₑ) / (1 − Pₑ)	(Pₐ − Pₑ) / (1 − Pₑ)
Chance term Pₑ	Product of the raters' marginals	2·q·(1 − q) for binary; (1 / (k − 1))·Σ πⱼ(1 − πⱼ) general
As one category gets rare	Pₑ climbs toward 1	Pₑ shrinks toward 0
Behavior under skew	Collapses (the paradox)	Stays stable (Gwet, 2008)
Measurement level	Nominal (weighted κ for ordinal)	Nominal via AC1; ordinal/interval via AC2

A worked example: 84% agreement, kappa 0.11, AC1 0.80

Here is the paradox in one synthetic table, and the two coefficients worked all the way through. Two coders each judge 100 utterances for a common feature—present or absent—that turns out to be present about 90% of the time.

	Coder B: present	Coder B: absent	Row total
Coder A: present	82	8	90
Coder A: absent	8	2	10
Column total	90	10	100

Observed agreement is the diagonal: Pₐ = Pₒ = (82 + 2) / 100 = 0.84. Both raters mark the feature present 90 times, so the prevalence of “present” is q = 0.90.

Cohen’s kappa corrects with a chance term from the marginals: Pₑ = 0.90·0.90 + 0.10·0.10 = 0.82, so κ = (0.84 − 0.82) / (1 − 0.82) = 0.02 / 0.18 = 0.11. Gwet’s AC1 uses the prevalence-stable chance term instead: Pₑ = 2·0.90·0.10 = 0.18, so AC1 = (0.84 − 0.18) / (1 − 0.18) = 0.66 / 0.82 ≈ 0.80.

By the widely quoted Landis and Koch (1977) bands, κ = 0.11 is only “slight” agreement, while AC1 ≈ 0.80 sits at the ceiling of the “substantial” band—a world apart, on identical data. Neither number is a lie; they answer the same question with different assumptions about how much of an 84% match two coders get for free when the base rate is 90%. That is why, under skew, you report both and let the gap between them tell the reader that prevalence is doing the work.

The second paradox: rater bias and marginal imbalance

The bias paradox is the mirror image of the prevalence one: when the two raters’ marginal totals disagree—one calls the feature present more often than the other—kappa is nudged higher for the same observed agreement, not lower (Feinstein & Cicchetti, 1990). Where prevalence deflates kappa, rater bias inflates it, and the two effects can partly mask each other in a single number.

This is the honest limit of AC1. It was built to neutralize the prevalence paradox, so it does not by itself resolve the bias paradox; large marginal asymmetry between raters still deserves scrutiny whichever coefficient you report. Cicchetti and Feinstein (1990) argued that no single omnibus index cleanly settles both problems, and recommended reporting the proportion of agreement on positive and on negative decisions separately, alongside the base rates. The practical takeaway is the same either way: never publish a lone coefficient without the marginals that produced it.

What is Gwet’s AC2, and when do you need weights?

Gwet’s AC2 is AC1 with disagreement weights, the paradox-resistant counterpart to weighted kappa for ordinal or interval ratings (Gwet, 2008; Gwet, 2014). AC1 counts any mismatch as a full disagreement, which is right for unordered categories and wrong for a graded scale, where rating an item “mild” when a colleague said “moderate” should cost less than “none” versus “severe.”

AC2 fixes that by weighting each disagreement by how far apart the two ratings are, the same intuition behind linear and quadratic weighted kappa, while keeping AC1’s prevalence-stable chance term underneath. That combination matters for clinician-rated severity scales, which are both ordinal and often skewed—most items sit near “absent” in a given interview. Andreasen’s Scale for the Assessment of Thought, Language and Communication (TLC) rates each disorder of speech by graded severity, and the Montgomery-Åsberg Depression Rating Scale (MADRS) produces an ordinal score per item; on either, a distance-weighted coefficient that also survives skew is the honest measure of how well two raters agree.

When should you use Gwet’s AC1 instead of kappa?

Reach for Gwet’s AC1 when one category dominates your labels, because that is precisely the condition that triggers the prevalence paradox and where kappa stops being trustworthy (Feinstein & Cicchetti, 1990). Rare-event annotation, screening for an uncommon phenomenon, and most yes/no transcript coding all live in this regime, so AC1 belongs in the default toolkit rather than the footnotes.

There is direct evidence behind the recommendation. In a nine-rater reliability study of DSM-IV personality-disorder ratings, Gwet’s AC1 “was also found to be less affected by prevalence and marginal probability than that of Cohen’s Kappa” and was recommended for inter-rater reliability analysis (Wongpakaran et al., 2013). Quarfoot and Levine (2016) reached a compatible conclusion for the many-rater case, showing that weighted, fixed-distance indices such as AC2 and the Brennan-Prediger coefficient hold up better than Fleiss’ kappa and Krippendorff’s alpha when units are unevenly spread across categories.

A word of caution keeps this balanced. Because AC1’s chance term goes quiet under extreme skew, it can read as generous exactly where the underlying agreement is genuinely thin, so a high AC1 on a rare label is not automatic proof of good coding. Treat it as one view of the data next to kappa and the marginals, not as the single number that ends the argument. For the full menu of coefficients and when each one fits, see our guide to which inter-rater reliability coefficient to use.

How do you compute Gwet’s AC1 in R and Python?

You will not hand-compute AC1 in production, and you don’t need to—Kilem Gwet’s own irrCAC package implements AC1 and AC2, with standard errors and confidence intervals, in both R and Python. Both reproduce the worked example above (AC1 ≈ 0.80) from the same 2×2 counts.

In R, gwet.ac1.table() takes the contingency table directly:

library(irrCAC)

# 2x2 counts: rows = Coder A, columns = Coder B (present, absent)
tbl <- as.table(matrix(c(82, 8,
                          8, 2), nrow = 2, byrow = TRUE))

gwet.ac1.table(tbl)$coeff.val   # 0.8049  (Gwet's AC1)

In Python, the irrCAC package builds a CAC object from a pandas contingency table and exposes gwet():

import pandas as pd
from irrCAC.table import CAC

# 2x2 counts: rows = Coder A, columns = Coder B (present, absent)
table = pd.DataFrame(
    [[82, 8],
     [8, 2]],
    index=["present", "absent"],
    columns=["present", "absent"],
)

cac = CAC(table)
print(round(cac.gwet()["est"]["coefficient_value"], 4))   # 0.8049

For ordinal severity data, switch to AC2 by passing weights (gwet.ac2.table(tbl, weights = "quadratic") in R, or a weights= argument to the CAC object in Python) so that near misses cost less than far ones, while the chance term stays paradox-resistant. The many-rater case, where you also weigh Krippendorff’s alpha against these coefficients, is worked in our tutorial on computing Krippendorff’s alpha.

Should you switch to Gwet’s AC1?

Switch to Gwet’s AC1 whenever your labels are lopsided, which in transcript coding and rare-event annotation is most of the time. The rule is not “AC1 beats kappa”; it is “under skewed prevalence, kappa alone misleads, so report AC1 (or AC2 for ordinal scales) beside it and let the two numbers, plus the base rates, describe the data honestly.” On balanced categories the coefficients agree and the choice barely matters; the paradox only bites when one category dominates.

In Tagaroo, each coding scheme runs as a live annotation workflow, and inter-rater reliability is computed as coders work, with the coefficient matched to the scale’s measurement level—an ordinal severity scale gets a distance-weighted statistic, a nominal present/absent label gets the paradox-aware form. If you want the fuller decision tree behind that matching, our guide to computing and interpreting Cohen’s kappa is the companion piece. Pick the chance correction for the base rate you actually have, not the one you wish you had: that is the difference between a number and a defensible reliability claim.

References

• Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1), 37–46. doi.org/10.1177/001316446002000104
• Landis, J. R., & Koch, G. G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33(1), 159–174. doi.org/10.2307/2529310
• Feinstein, A. R., & Cicchetti, D. V. (1990). High agreement but low kappa: I. The problems of two paradoxes. Journal of Clinical Epidemiology, 43(6), 543–549. doi.org/10.1016/0895-4356(90)90158-L
• Cicchetti, D. V., & Feinstein, A. R. (1990). High agreement but low kappa: II. Resolving the paradoxes. Journal of Clinical Epidemiology, 43(6), 551–558. doi.org/10.1016/0895-4356(90)90159-M
• Gwet, K. L. (2008). Computing inter-rater reliability and its variance in the presence of high agreement. British Journal of Mathematical and Statistical Psychology, 61(1), 29–48. doi.org/10.1348/000711006X126600
• Wongpakaran, N., Wongpakaran, T., Wedding, D., & Gwet, K. L. (2013). A comparison of Cohen’s Kappa and Gwet’s AC1 when calculating inter-rater reliability coefficients: a study conducted with personality disorder samples. BMC Medical Research Methodology, 13, 61. doi.org/10.1186/1471-2288-13-61
• Gwet, K. L. (2014). Handbook of Inter-Rater Reliability (4th ed.). Advanced Analytics, LLC.
• Quarfoot, D., & Levine, R. A. (2016). How robust are multirater interrater reliability indices to changes in frequency distribution? The American Statistician, 70(4), 373–384. doi.org/10.1080/00031305.2016.1141708