Kappa Statistic Calculator

Cohen's Kappa (κ) is a statistical measure of inter-rater reliability that accounts for the agreement expected purely by chance. Simple percent agreement ignores the fact that two raters could agree on some items randomly. Kappa corrects for this, providing a more accurate picture of genuine consensus between raters.

The Landis & Koch (1977) guidelines are the most widely used: κ ≤ 0 indicates no agreement, 0.01–0.20 is slight, 0.21–0.40 is fair, 0.41–0.60 is moderate, 0.61–0.80 is substantial, and 0.81–1.00 is almost perfect agreement. A kappa of 1.0 means perfect agreement; 0 means agreement no better than chance.

The confusion matrix (also called a contingency table) holds counts of how often each rater assigned each category. Each cell [Row i, Col j] contains the number of subjects that Rater 1 placed in category i and Rater 2 placed in category j. Diagonal cells represent agreements; off-diagonal cells represent disagreements.

This calculator supports 2 to 5 rating categories. Select the number of categories in the setup dropdown, then fill in the corresponding cells of the confusion matrix. Cells outside your selected matrix size are ignored in the calculation.

Kappa is calculated as κ = (Po − Pe) / (1 − Pe), where Po is the observed proportion of agreement (sum of diagonal cells divided by total N) and Pe is the expected agreement by chance (computed from the row and column marginal totals). The standard error is then used to construct the z-score and confidence interval.

A negative kappa indicates that the two raters agree less than would be expected by chance alone. This is rare in practice and often suggests a systematic disagreement — for example, one rater tends to consistently choose the opposite category from the other rater.

Weighted Kappa is preferred when the rating categories are ordered (ordinal), because it gives partial credit for near-misses. For example, if categories are 'None', 'Mild', 'Moderate', and 'Severe', disagreeing by one level should count less than disagreeing by three levels. For nominal (unordered) categories, use unweighted Kappa as provided here.

There is no universal rule, but most researchers recommend at least 30–50 subjects per category for a stable Kappa estimate. Very small samples produce wide confidence intervals, making the kappa value unreliable. The p-value and 95% CI shown by this calculator will reflect the uncertainty caused by small sample sizes.