Evaluating pilot results

Learning objectives

• Explain the purpose of evaluating pilot results in a garment-factory improvement project.
• Apply the described method to a representative shop-floor situation.
• Recognize the common mistakes and how to avoid them.

Concept

Pilot evaluation combines statistical evidence, practical effect, process stability, risk, and implementation learning.

Plot time order first, check measurement comparability, then calculate effect and confidence interval.

Select paired, independent, or proportion test as appropriate; guardrail metrics matter as much as the primary Y.

Variables, units, and assumptions

Primary Y (defects/unit or minutes), guardrails (output/hr, ergonomic score), period (dates).

Comparable measurement, similar style mix pre/post, no confounding change.

Free-spreadsheet workflow

Free spreadsheet: paste pre/post data, plot run chart, compute mean/SD, 95% CI on difference (t-based), stratify by shift.

Interpretation and limitations

Effect exceeds CI half-width AND meets practical threshold AND passes guardrails → confirm.

Short pilots may miss seasonality; guardrails may reveal delayed effects.

Garment-factory example

Compare FPY and output per hour on pilot and comparable lines while stratifying by style and shift.

Method

1. Verify protocol fidelity.
2. Clean data.
3. Graph in time order.
4. Estimate effect and CI.
5. Test assumptions.
6. Calculate financial impact.
7. Summarize operational lessons.

Common mistakes

• Relying only on p-value while ignoring practical effect.
• Ignoring negative effects on output or ergonomics.