How this calculation works
Two raters can agree often simply because they both use one category frequently. Cohen’s kappa compares observed agreement with agreement expected from their separate category margins. It therefore answers a different question from raw percentage agreement. This page supports exact-category agreement and linear or quadratic weights for ordered categories. The paired count matrix remains visible through its marginal summaries so the denominator can be inspected.
kappa = (observed weighted agreement − marginal expected agreement)/(1−marginal expected agreement)A worked example
In a two-category table with diagonal counts forty and forty and off-diagonal counts ten and ten, observed agreement is 0.8. Each rater uses each category half the time, giving expected agreement 0.5. Kappa is therefore (0.8−0.5)/(1−0.5), or 0.6. Perfect diagonal agreement across several used categories gives one under each supported weight convention.
Good to know
Only two raters and complete paired categories are supported. Missing ratings, multiple-rater agreement and repeated dependent ratings require other methods. No confidence interval, hypothesis test or quality-certification decision is returned. The result needs the category definitions and the study design to be interpreted responsibly.
Reference: scikit-learn — Cohen kappa and weights (opens in a new tab) ↗
What this calculation answers
Two raters can agree often simply because they both use one category frequently. Cohen’s kappa compares observed agreement with agreement expected from their separate category margins. It therefore answers a different question from raw percentage agreement.
This page supports exact-category agreement and linear or quadratic weights for ordered categories. The paired count matrix remains visible through its marginal summaries so the denominator can be inspected.
The Binary confusion matrix precision recall and F1 calculator evaluates labelled predictions against a reference outcome. Two-rater agreement does not automatically have such a truth reference.
Use the right input conventions
Rows represent rater A and columns rater B using the same category order. Counts must be whole nonnegative numbers in a square table. The diagonal contains exact category matches.
Linear and quadratic weights depend on category-index distance, not a separate user-supplied physical distance between labels. Do not use them for nominal categories unless that ordering has a defensible meaning.
Interpret the result
Kappa is not accuracy against an objective truth. Both raters can agree and still be wrong. It is also sensitive to prevalence and marginal distributions, so one universal interpretation cutoff is not supplied.
If expected agreement is one, the denominator is zero and kappa is undefined. The page rejects that degenerate table rather than assigning an apparently perfect or zero coefficient.
Work through the example
In a two-category table with diagonal counts forty and forty and off-diagonal counts ten and ten, observed agreement is 0.8. Each rater uses each category half the time, giving expected agreement 0.5.
Kappa is therefore (0.8−0.5)/(1−0.5), or 0.6. Perfect diagonal agreement across several used categories gives one under each supported weight convention.
Where the model stops
Only two raters and complete paired categories are supported. Missing ratings, multiple-rater agreement and repeated dependent ratings require other methods.
No confidence interval, hypothesis test or quality-certification decision is returned. The result needs the category definitions and the study design to be interpreted responsibly.
A few questions, answered
Is high kappa evidence that the ratings are correct?
No. It measures agreement between raters after the marginal correction, not correctness against an external truth.
What changes in the comparison?
Use three ordered categories and inspect quadratic weighting of adjacent disagreements.