Attribute agreement analysis

Learning objectives

• Design an attribute agreement study for categorical decisions.
• Calculate within-, between- and standard-agreement rates.
• Investigate false-accept and false-reject risk at the boundary.

Where agreement matters

Attribute agreement evaluates whether inspectors classify the same items the same way and match an expert standard, especially near the acceptance boundary.

Percent agreement can look high when one category dominates; report agreement with the standard and use kappa cautiously.

Garment-factory example

Inspectors classify seam appearance as acceptable/unacceptable across a coded sample set including borderline cases, rating each item twice without seeing previous ratings.

Method

1. Prepare a reference-graded sample set including boundary cases.
2. Blind and randomise; rate each item twice.
3. Cross-tabulate and calculate agreement rates.
4. Investigate disagreements at the boundary.

Common mistakes

• Using only obvious defects.
• Reporting overall percent agreement when one category is 90% of the set.