Oliver S. Crocco

Tools

Inter-Rater Reliability Calculator

Krippendorff's alpha, Fleiss's kappa, Cohen's kappa, and percent agreement for a coding team of any size, on any number of items, with any set of codes. Blank cells are handled, every statistic updates as you type, and the items that need adjudication are sorted for you.

Private by design.This runs entirely in your browser. Nothing you enter is sent anywhere, so student and participant data stay on your machine. The grid is kept in this browser's local storage so a refresh does not lose it; Clear removes it.

1 Set up

2 Enter the codes

Type a code in each cell. A blank cell means that rater did not code the item, which Krippendorff's alpha tolerates and Fleiss's kappa does not. Enter and the arrow keys move around the grid; Tab moves across. Pasting a block from a spreadsheet into a cell fills the grid from that cell. Rater names can be edited in place.

3 Results

Each code on its own

Alpha for one code against all the others shows where the disagreement sits. A low value here next to an acceptable overall alpha means one distinction is unstable while the rest hold.

Between pairs of raters

Cohen's kappa for every pair, with each rater's mean across their pairs. A rater whose mean sits well below the others is worth a conversation before the codebook is blamed.

Item by item

The share of rater pairs that agreed on each item, with the count of each code. Items below 60% are outlined.

Adjudication queue

Lowest agreement first. Work down the list; the items near the top show what the codebook does not yet say.

    About these statistics

    When a reliability coefficient is the right instrument

    What decides whether these numbers mean anything is what the codes are for. When codes function as measurement, so that a claim depends on the code having been applied the same way regardless of who applied it, reliability is the evidence for that claim. Systematic review screening, data extraction, and risk-of-bias assessment work this way, and reporting standards expect a coefficient. So does quantitative content analysis in the tradition Krippendorff (2019) writes from, along with codebook approaches such as framework and template analysis, directed content analysis (Hsieh & Shannon, 2005), structured observation, and mixed methods designs where codes become variables in a later analysis.

    Codes do other work elsewhere. In reflexive thematic analysis, coding is part of how the researcher thinks, and themes are actively generated by a situated analyst. Braun and Clarke (2019, 2021) argue that measuring agreement in that setting imports assumptions from a paradigm the method has left behind, and that requiring consensus pulls analysis toward the shallowest reading two people can share. Interpretative phenomenological analysis and much constructivist grounded theory take a similar position. Work in those traditions is judged by other means, including the trustworthiness criteria in Lincoln and Guba's tradition, an audit trail, reflexive documentation of the analyst's position, and accounts of how the interpretation developed. A coefficient from this page speaks to none of that.

    The debate itself is still live, and McDonald, Schoenebeck, and Forte (2019) is a useful map of where the expectation of a coefficient comes from and where it does not belong. Two further points are worth keeping in view. A high coefficient reports consistency and says nothing about whether a scheme is any good, since a scheme can be reliably applied and still describe the wrong things. And multiple coders remain valuable in interpretive work, with the difference that divergence there becomes material for discussion. The per-item view and the adjudication queue on this page can be read that way, as a means of finding where readings parted company, with the coefficients left aside.

    Krippendorff's alpha is the headline figure. It works for any number of raters, tolerates blank cells, and can treat the codes as unordered categories, ordered categories, or numbers on an interval scale. The calculator computes it from the coincidence matrix as Krippendorff (2011) sets out and reports a 95% confidence interval from 2,000 bootstrap resamples of the items, drawn with a fixed seed so the same data always give the same interval (Zapf et al., 2016). Krippendorff (2019) suggests treating .800 as the level at which coded data can carry conclusions and .667 as the floor for tentative ones. Those are conventions from content analysis. The level that matters is the one your field and your claims can defend, and a scheme used to describe a small corpus does not carry the same burden as one used to test a hypothesis.

    Fleiss's kappa is shown beside alpha for readers who expect it. It requires every item to be coded by the same number of raters and ignores which rater gave which code, so when the number of raters differs from item to item it is left blank rather than estimated from a subset. It treats the codes as unordered whatever level of measurement is selected.

    Cohen's kappa compares two raters at a time and appears in three places: between every pair of raters, for each rater against the reference row, and for the most common code across raters against the reference row. The comparison with a reference is a check on an existing coding, not a validity claim. Where most of the independent raters converge against the reference, the reference deserves a second look. Landis and Koch's (1977) descriptive labels for kappa are often quoted; their authors offered them as arbitrary divisions, and they are read most safely that way.

    Mean pairwise agreement is the share of rater pairs that agreed, averaged over items, with no correction for chance. It is always the most flattering number and is shown so it can be read next to the chance-corrected ones. High agreement with a low coefficient usually means one code dominates the data (Feinstein & Cicchetti, 1990), so check how often each code was used before deciding what a low coefficient means.

    The per-code alpha treats one code against all the others, which shows whether disagreement is spread across the scheme or concentrated in one distinction. The item view and the adjudication queue sort items by their pairwise agreement so a team can discuss the hardest cases first.

    How the calculations were checked

    The estimators reproduce hand-derived cases and the worked example in Krippendorff (2011). On 240 randomly generated grids with two to eight raters, with and without missing values, they match the R package irr and the Python package krippendorff to fourteen decimal places and Gwet's irrCAC package to the five decimals it reports: alpha at all three levels, Fleiss's kappa, Cohen's kappa for every pair of raters, percent agreement, and the per-code alpha. One caution for readers who use R: irr's kripp.alpha (version 0.84.1) mishandles grids with no blank cells and three or more raters, and the comparison used a workaround that forces its general code path. The verification scripts and that finding are documented in the calculator's repository at github.com/olivercrocco/irr-calculator.

    Privacy

    Everything runs in your browser. Nothing you enter is sent anywhere. The grid is kept in this browser's local storage so a refresh does not lose it, and Clear removes it.

    Citing the calculator

    Crocco, O. S. (2026). Inter-rater reliability calculator (Version 1.0.0) [Computer software]. https://olivercrocco.com/irr-calculator

    Funding

    This calculator came out of the redesign of LHRD 7923, Advanced Mixed Methods Research, supported by a Course Integration Award from the AI Workforce Transformation Initiative of the School of Leadership & Human Resource Development at Louisiana State University.

    References