Measurement-system resolution and accuracy

Learning objectives

• Distinguish resolution from accuracy in a measurement system.
• Detect bias against a traceable reference.
• Separate instrument error from process variation.

Resolution and accuracy

Resolution is the smallest increment the instrument can display. Accuracy is closeness to a traceable reference. A digital display can be precise but inaccurate; a coarse ruler may be accurate but useless for tight tolerances.

Rule of thumb: resolution should be 10× tighter than the tolerance or process spread it must judge.

Garment-factory example

A ruler marked to 1 cm cannot support a ±3 mm tolerance. A stretched fabric tape may repeat itself well but read low by 5 mm against a steel reference.

Method

1. Compare instrument resolution to the tolerance and process spread.
2. Measure a traceable reference repeatedly.
3. Calculate average bias and compare to the tolerance.
4. Correct, calibrate or replace the system.

Common mistakes

• Assuming a digital display implies accuracy.
• Calibrating in the lab but ignoring the shop-floor method.