How to Measure Agreement When Analysts Classify the Same Business Records Differently

Two analysts can review the same supplier, customer, product, lead, or transaction record and reach different conclusions. One marks a supplier “eligible”; another sends it to “needs review.” One assigns a support case priority 1; another selects priority 3.

That is more than an individual performance issue. If the same documented process produces different labels depending on who applies it, downstream counts, queues, and decisions may depend on the reviewer rather than the record.

The useful question is not only, “What is our kappa?” It is: which records and category boundaries create disagreement, why, and is that variation acceptable for the decision the labels will drive?

Start with the classification task

Agreement is a test of repeatability, not accuracy. Analysts may agree perfectly on a rule that lacks the evidence needed for a sound decision. Conversely, disagreement may reveal a vague definition, missing source information, or a category scheme that does not fit the records being reviewed.

Before measuring anything, define the task. Specify the unit being classified, the allowed categories, the evidence analysts may use, and what to do when information is absent or conflicting. State whether categories are mutually exclusive and whether more than one label is permitted.

An “unknown” or “needs review” option also needs a rule. Without one, some analysts will make a best guess while others will escalate the same uncertainty. The resulting disagreement says as much about the workflow as it does about the people using it.

Use an independent common sample

Have two or more analysts classify the same records independently and without seeing one another’s decisions. Discussion and adjudication are useful later, but they should not occur before the initial measurement.

The sample should include ordinary records, because those represent normal operations. It should also deliberately include rare, ambiguous, high-value, and decision-critical cases. A review made up entirely of easy records can produce reassuring agreement while missing the boundary that creates costly errors.

Keep the sample design visible. A deliberately difficult set is valuable for improving the rulebook, but it may not estimate agreement in day-to-day production. When practical, report those uses separately.

Read the disagreement matrix before the headline statistic

Put one analyst’s categories in rows and the other’s in columns. The diagonal cells are matches. Off-diagonal cells show the specific conflicts.

Suppose two analysts independently classify 100 suppliers as eligible, needs review, or ineligible. They agree on 84 records, producing 84% observed agreement. That is a useful, plain-language number.

But the matrix may show that 9 of the 16 mismatches are between eligible and needs review. That finding is more actionable than the overall percentage. It suggests that the evidence rule separating automatic approval from manual review needs clearer examples or a better decision rule.

A mismatch between reporting subtypes may be tolerable. A mismatch between eligible and ineligible may reverse an action. Overall agreement can conceal that distinction, so inspect category-level patterns before declaring the process reliable.

Choose a statistic that fits the design

Observed agreement is the share of records on which analysts assigned the same label. It is directly understandable, but it does not account for agreement expected under a chance model.

For two analysts assigning unordered categories, Cohen’s kappa is a conventional chance-corrected summary. For more than two analysts with the same number of ratings per record, Fleiss’ kappa is a common extension.

If categories have a meaningful order, weighted kappa may fit better. Consider support priorities of 1, 2, and 3: a split between 2 and 3 may be less consequential than a split between 1 and 3. The weighting should reflect documented business meaning, not a default mathematical preference.

Krippendorff’s alpha is another flexible option, particularly when there are multiple analysts, some missing ratings, or measurement levels beyond nominal categories.

No coefficient is universally best. The choice depends on the number of analysts, category type, missing ratings, sampling design, and the consequences of disagreement.

Do not let kappa stand alone

Always report observed agreement, category frequencies, and the cross-tabulation alongside a chance-corrected statistic. Kappa can behave unexpectedly when one category is overwhelmingly common or analysts use categories at different rates.

For example, if 97% of records are routinely labeled “standard,” analysts may agree on nearly every record while kappa is lower than stakeholders expect. That does not make observed agreement meaningless or kappa defective. It is a prompt to examine the category distribution and the small minority of records where the business risk may be concentrated.

A small common sample also produces an unstable headline number. Where the statistic will influence a meaningful decision, report an uncertainty interval and avoid overreading small changes between review rounds.

Turn disagreement into process improvement

Review the largest off-diagonal cells first. Look at the underlying records and distinguish among several causes: unclear rules, incomplete evidence, inconsistent treatment of missing information, unsuitable categories, and training gaps.

Then make a specific change. Add examples to the rulebook. Clarify the evidence needed for a label. Create an exception path for records that cannot be resolved from available information. If categories are too broad or too artificial, reconsider the scheme itself.

Retest on fresh records after the update. Reusing the same examples can show whether people remember the resolution, rather than whether the process became more repeatable.

A compact report can keep this work honest: describe the sample and rulebook version; show category shares and the agreement matrix; report observed agreement and the chosen reliability statistic with uncertainty; then name the major disagreement patterns and the changes they prompted.

The goal is not to chase a universal cutoff. It is to establish that classifications are repeatable enough for their intended use, while making the remaining uncertainty visible where it matters most.