8. DMAIC: Improve and Experimental Design

Introduction and Context

The Improve phase of DMAIC converts analytical knowledge into better process performance. In the preceding Define, Measure, and Analyze phases, the improvement team has clarified the problem, established reliable measures, identified performance gaps, and investigated probable causes. Improve begins when the team can answer an important question: which changes are most likely to remove or control the causes of poor performance without creating unacceptable cost, risk, or operational disruption?

Improvement is not synonymous with generating a long list of ideas. It is a disciplined process of designing, evaluating, testing, and selecting interventions. A technically attractive solution may be difficult to implement, vulnerable to human error, expensive to maintain, or harmful to another quality characteristic. Consequently, Six Sigma improvement combines creativity with evidence, experimentation with managerial judgment, and optimization with risk control.

Experimental design is central to advanced Improve work. Rather than changing one factor at a time, design of experiments (DOE) enables teams to study several factors simultaneously and to identify interactions among them. A well-designed experiment can reveal not only which input variables matter, but also which combination of settings produces a robust and economically desirable response.

Learning Objectives

After studying this chapter, the learner should be able to:

1. Explain the purpose, logic, and outputs of the Improve phase of DMAIC.

2. Generate improvement alternatives using creativity, benchmarking, and process knowledge.

3. Prioritize solutions through feasibility, risk-benefit, cost, and impact analysis.

4. Design and interpret pilot studies and factorial experiments.

5. Distinguish main effects from interaction effects and explain response analysis.

6. Select operating settings that balance performance, robustness, cost, and risk.

7. Develop an implementation plan that connects tested improvements with Control-phase requirements.

8. Evaluate ethical, technical, organizational, and digital considerations in improvement projects.

From Diagnosed Causes to Designed Solutions

The Improve phase should begin with a verified cause-and-effect model rather than with untested opinions. A cause identified during Analyze becomes a useful improvement target only when evidence supports its relationship with the critical-to-quality (CTQ) characteristic. The team should also determine whether the cause is controllable, whether its effect is practically important, and whether changing it may create adverse side effects.

A useful distinction is between a solution and a countermeasure. A solution is a proposed response to a problem; a countermeasure is a deliberately designed intervention linked to a verified cause. For example, “provide more training” is a broad solution. If analysis shows that errors occur because a particular decision rule is ambiguous, a more precise countermeasure may be to redesign the rule, add an electronic prompt, and test whether the intervention reduces error without increasing cycle time.

Improvement alternatives commonly address four areas:

• elimination of non-value-adding activities;
• reduction of variation in materials, methods, machines, or measurement systems;
• prevention or early detection of defects; and
• redesign of the process, product, information flow, or work environment.

The best intervention is not always the one that produces the largest average improvement. A sustainable solution should also be understandable, maintainable, compatible with regulations, and resilient under normal operating variation.

Generating and Selecting Improvement Alternatives

Structured creativity

Creativity in Six Sigma is more effective when it is structured around the process and the verified causes. Brainstorming, affinity diagrams, morphological analysis, SCAMPER, design thinking, and mistake-proofing techniques can be used to generate alternatives. Cross-functional participation is valuable because operators, engineers, customers, maintenance staff, suppliers, and compliance personnel observe different forms of process failure.

Brainstorming should initially separate idea generation from evaluation. Premature criticism suppresses unconventional possibilities. Once alternatives have been generated, however, disciplined screening is required. Each idea should be connected to a cause, an expected mechanism of improvement, and an observable response.

Benchmarking and external learning

Benchmarking involves comparing processes, practices, or performance with relevant internal or external references. It is not simply copying another organization. A benchmark must be interpreted in context: differences in technology, demand, product mix, regulation, workforce capability, and customer requirements may make direct transfer inappropriate.

Useful benchmarking questions include:

• What process characteristic is being compared?
• Is the benchmark based on outcomes, methods, or both?
• Which conditions make the benchmark successful?
• What would have to change for the practice to work in the focal process?
• Could adopting the practice create legal, ethical, or intellectual-property concerns?

Prioritization through decision criteria

A project team can use a weighted decision matrix to compare alternatives. Criteria may include expected CTQ improvement, implementation cost, time to benefit, technical feasibility, customer impact, operational disruption, regulatory exposure, and sustainability. Each alternative receives a score against each criterion, and the scores are multiplied by criterion weights.

The matrix is a decision aid, not a substitute for judgment. Scores should be supported by evidence, assumptions should be recorded, and sensitivity analysis should be performed. If a small change in weights reverses the ranking, management should recognize that the decision is uncertain rather than treating the highest numerical score as definitive.

Risk-benefit analysis

Risk-benefit analysis examines both the expected gains and possible adverse consequences of a proposed change. Failure Mode and Effects Analysis (FMEA) is especially useful. For each potential failure mode, the team considers severity, occurrence, and detectability, together with existing controls. Traditional risk priority numbers can assist comparison, but they should not be treated as mathematically precise measures of risk. A high-severity failure may require action even when occurrence is low.

A proposed improvement should be assessed for risks such as:

• new failure modes introduced by the change;
• increased dependence on a single machine, supplier, or employee;
• cybersecurity or data-integrity exposure;
• unsafe work conditions or ergonomic burden;
• unintended effects on another CTQ;
• noncompliance with contractual, statutory, or industry requirements; and
• difficulty reverting to the previous process if the pilot fails.

Pilot Studies: Learning Before Full-Scale Deployment

A pilot is a controlled, limited implementation used to assess whether a proposed change works in the intended environment. It differs from an informal trial because its objectives, measures, duration, responsibilities, and decision rules are specified in advance.

A sound pilot plan identifies the population or process segment, baseline performance, intervention, comparison condition where feasible, sample size or observation period, CTQs, potential confounders, and criteria for continuation or termination. The team should determine whether the pilot is intended primarily to test technical feasibility, estimate performance improvement, assess user acceptance, or evaluate operational readiness.

Pilot results must be interpreted cautiously. Improvement may arise from temporary attention, unusual demand, selective participation, or the Hawthorne effect. Therefore, teams should collect data under representative operating conditions and examine stability over time. A pilot that produces a favorable average but creates excessive variation or unacceptable failures should not be scaled without redesign.

The pilot should conclude with a documented decision: adopt, adapt, repeat with stronger controls, or abandon. A successful pilot is not automatically a permanent solution; it is evidence supporting a controlled implementation decision.

Design of Experiments as a Six Sigma Method

Core terminology

In DOE, the response is the output measured by the experiment, such as defect rate, tensile strength, cycle time, or customer waiting time. Factors are input variables that may influence the response. Levels are the settings selected for each factor. A treatment or experimental run is a particular combination of factor levels. Replicates are repeated runs under the same treatment, while randomization determines the order of runs to reduce bias from time-related influences. Blocking groups similar experimental conditions, such as production batches or shifts, to separate nuisance variation from factor effects.

The experimental region is the range of factor settings studied. Results should not be extrapolated confidently beyond that region unless supported by additional knowledge. The design must also distinguish controllable factors from noise factors. Noise factors cannot always be eliminated, but their influence can be measured or deliberately incorporated into a robust design.

Why one-factor-at-a-time experimentation is inadequate

Changing one factor while holding all others constant appears simple, but it has serious limitations. It may require many runs, miss interactions, and produce misleading conclusions when the effect of one factor depends on the level of another. DOE varies factors systematically so that their individual and joint effects can be estimated efficiently.

Full factorial designs

A two-level full factorial design with k factors requires 2^k combinations. For two factors, A and B, the four combinations are low-low, high-low, low-high, and high-high. The main effect of A is the change in the average response when A moves from its low to high level, averaged across levels of B. The main effect of B is defined similarly.

An interaction exists when the effect of one factor changes according to the level of another. Graphically, nonparallel lines in an interaction plot suggest interaction; crossing lines often indicate a particularly strong interaction. When interaction is important, interpreting main effects in isolation can be deceptive. The team should focus on the combinations of settings rather than selecting each factor independently.

Full factorial designs are attractive because they provide complete information within the chosen region, but the number of runs increases rapidly as factors are added. Replication helps estimate pure experimental error and assess repeatability, while randomization protects against time trends and hidden biases.

Fractional factorial designs and screening

When many factors are suspected, a fractional factorial design uses a carefully selected fraction of the full set of combinations. It reduces the number of runs but introduces aliasing, meaning that some effects cannot be separated completely. A fractional design is therefore most suitable for screening, particularly when higher-order interactions are believed to be small. Once the important factors have been identified, a more detailed design may be used for optimization.

Analysis of experimental results

Analysis generally includes graphical examination, estimates of main and interaction effects, analysis of variance (ANOVA), residual analysis, and practical interpretation. Statistical significance should not be confused with managerial significance. A very small effect may be statistically detectable in a large experiment but irrelevant economically. Conversely, an important effect may fail to reach statistical significance in a small or noisy experiment.

Residuals should be examined for non-random patterns, unequal variance, non-normality where relevant, and outliers. An apparently impressive model may be unreliable if its assumptions are violated. The team should also check whether the measured response is trustworthy and whether the experimental runs were conducted according to the plan.

Response optimization and robustness

Response analysis uses a fitted model to predict performance across the experimental region. Contour plots, interaction plots, response surfaces, and desirability functions can help identify settings that meet several objectives simultaneously. For example, a manufacturing team may seek to minimize defects and cycle time while maintaining strength above a specified threshold.

Optimization should consider robustness. A setting that produces the best result only under ideal conditions may be inferior to a slightly less optimal setting that is insensitive to ordinary noise. Robust design therefore seeks operating conditions that deliver acceptable performance across realistic environmental, material, equipment, and operator variation.

An Integrated Improve Workflow

A practical Improve workflow can be organized as follows:

1. Translate verified causes into improvement hypotheses.

2. Generate alternatives through structured creativity and relevant benchmarking.

3. Screen alternatives for feasibility, risk, cost, ethics, and customer impact.

4. Define the desired response and select measurable CTQs.

5. Conduct a pilot or screening experiment where uncertainty is high.

6. Use factorial DOE to estimate main effects and interactions.

7. Analyze responses statistically and operationally.

8. Select settings using performance, robustness, economics, and risk criteria.

9. Validate the selected solution under representative conditions.

10. Prepare implementation and Control-phase requirements.

This sequence is not rigid. A low-risk process change may proceed through a small pilot, whereas a high-risk change may require formal validation, regulatory review, and staged deployment. The central principle is proportionality: the effort devoted to testing and governance should reflect the consequences of failure.

Implementation Planning and Transition to Control

Implementation planning converts an experimental result into a managed process change. The plan should identify the process owner, activities, resources, timetable, training, communication, documentation, equipment modifications, supplier implications, and acceptance criteria. It should also specify how the old and new methods will be distinguished during transition.

A change-control procedure is essential where the process affects product safety, financial reporting, regulated records, or contractual obligations. Standard operating procedures, work instructions, process maps, and control plans must be revised. Measurement systems should be checked again if the improvement changes the characteristic being measured or the method of measurement.

Implementation should include a scale-up strategy. Possible approaches include phased rollout, parallel operation, pilot-site expansion, or a controlled cutover. Early implementation data should be reviewed against predicted results. If performance deteriorates, predefined escalation rules should trigger investigation rather than informal adjustment.

The Control phase receives the output of Improve: validated operating settings, control limits or reaction rules where appropriate, monitoring responsibilities, visual management, mistake-proofing, audit arrangements, and a response plan for drift. A solution is incomplete if the organization cannot detect deterioration or maintain the new standard.

Illustrative Case: Reducing Laboratory Sample-Preparation Errors

Background

A fictional analytical laboratory serving a pharmaceutical manufacturing site experienced repeated sample-preparation errors. The CTQ was the proportion of samples requiring rework. Analyze-phase work showed that errors were concentrated in dilution calculations and in the transfer of sample identifiers between paper worksheets and the laboratory information system.

Situation and analysis

The team verified that error frequency increased during peak workload periods. Interviews suggested that training alone had not resolved the issue. A cause analysis identified three potentially controllable factors: worksheet design, electronic prompts, and batch size. The team also recognized a risk that stronger checking controls could increase preparation time.

Improvement alternatives

The team generated four alternatives: redesign the worksheet, introduce barcode scanning, add an electronic dilution calculator, and require a second-person check for every sample. Benchmarking showed that barcode identification and electronic calculation were used successfully in comparable laboratories, but local data-integrity and validation requirements had to be considered.

A weighted decision matrix favored a combined intervention: barcode identification plus a validated electronic calculator, supported by a redesigned exception-based review rather than universal second-person checking. FMEA identified a new risk: an incorrect software formula could systematically affect many samples. Software validation and independent formula verification were therefore included in the pilot plan.

Pilot and experiment

The laboratory conducted a limited pilot during representative shifts. The response measures were rework rate, preparation time, and user-reported difficulty. A two-level factorial experiment then examined three factors: barcode use, electronic calculation, and batch size. The experiment revealed strong main effects for barcode use and electronic calculation. It also showed an interaction: electronic calculation produced the greatest reduction in errors when barcode identification was active, because correct data were transferred into the calculation interface. Large batch size increased preparation time variability when neither digital control was used.

Residual and stability checks indicated that the improvement was not limited to one shift. However, the team found that the first version of the electronic interface generated confusion when an exception occurred. The interface was redesigned before expansion.

Outcome and implementation

The selected process combined barcode scanning, a validated calculation tool, exception-based review, and revised work instructions. Implementation was phased across laboratory teams. Supervisors monitored rework, cycle time, exception frequency, and data-integrity events. The Control plan specified software change control, periodic formula verification, user access control, and a reaction procedure for unexpected error patterns.

Learning points

The case illustrates several principles. First, a countermeasure should address verified causes rather than symptoms. Second, DOE can reveal an interaction that would be missed by changing one factor at a time. Third, a reduction in defects is insufficient if the solution creates unacceptable time or compliance risks. Finally, digital improvement requires governance, validation, and monitoring rather than assuming that automation is automatically reliable.

Challenges, Risks, and Common Mistakes

Improve projects often fail because teams move to solutions before confirming causes. Other common errors include using vague responses, selecting factor levels that are too narrow, ignoring interactions, conducting runs in a non-random order, and treating a pilot as proof of universal effectiveness. Poor measurement systems can make an experiment appear decisive when the observed change is measurement noise.

Teams may also overfit experimental data, especially when many factors and responses are examined without a clear analytical plan. Statistical software can produce attractive plots and models, but it cannot correct biased sampling, confounding, missing data, or invalid assumptions. Practical validation remains necessary.

Organizational risks are equally important. Operators may resist a change imposed without participation; managers may demand rapid implementation before uncertainty is resolved; and incentive systems may reward local speed at the expense of total process quality. Improvement leaders should make assumptions visible, involve affected employees, and distinguish evidence from preference.

Ethical, Governance, and Professional Considerations

Improvement decisions should protect customers, employees, communities, and the integrity of organizational information. Teams must not suppress unfavorable pilot results, selectively report favorable runs, or manipulate factor settings to obtain a desired conclusion. Data exclusions require documented justification.

Where experiments involve people, privacy, safety, workload, or employment conditions, appropriate authorization and communication are required. In regulated industries, validation, traceability, document control, and formal change approval may be mandatory. Automated decisions should be explainable enough for responsible review, and access to operational data should follow least-privilege principles.

Digital Tools and Emerging Developments

Modern DOE is supported by statistical software, manufacturing execution systems, laboratory information systems, process mining, simulation, digital twins, and real-time dashboards. These technologies can reduce data-collection effort and enable more frequent experimentation. Machine learning may assist in identifying nonlinear relationships or promising experimental regions, but predictive accuracy does not establish causality. Designed experimentation remains necessary when the purpose is to learn what will happen under deliberate intervention.

Adaptive experimentation, Bayesian optimization, and sequential DOE are increasingly useful when experiments are expensive or when learning can occur in stages. Their use requires strong governance because algorithms may recommend settings outside validated operating boundaries. The future of Improve is therefore likely to combine statistical experimentation, domain expertise, simulation, and digital monitoring while retaining disciplined human accountability.

Best Practices for Advanced Improve Work

Effective practice includes the following:

• link every proposed intervention to a verified cause;
• define responses, constraints, and practical significance before testing;
• involve process owners and users in solution design;
• use pilots to learn under realistic conditions;
• apply factorial designs when interactions may matter;
• randomize and block experiments where appropriate;
• distinguish statistical significance from business significance;
• optimize for robustness, not merely the best average result;
• document assumptions, exclusions, risks, and decision rules;
• validate digital and regulated changes formally; and
• transfer explicit controls, ownership, and reaction plans to the Control phase.

Chapter Summary

The Improve phase transforms verified causes into tested and sustainable countermeasures. It combines structured creativity, benchmarking, decision matrices, FMEA, pilot studies, and design of experiments. Full factorial designs estimate main effects and interactions, while fractional designs support efficient screening. Response analysis and optimization must consider statistical evidence, practical impact, cost, risk, and robustness. A successful improvement is validated under representative conditions, implemented through controlled change management, and transferred to the Control phase with clear ownership and monitoring. Ethical reporting, regulatory compliance, digital governance, and employee involvement are essential to professional Six Sigma practice.

Key Terms

Improve phase: The DMAIC stage in which solutions are designed, tested, selected, and implemented.

Countermeasure: A targeted intervention intended to address a verified process cause.

Pilot study: A limited and controlled implementation used to evaluate feasibility and likely effects.

Design of experiments (DOE): A structured method for studying the effects of multiple factors on one or more responses.

Response: The measured output or performance characteristic examined in an experiment.

Factor: An input variable deliberately varied or studied in an experiment.

Level: A specific setting or value of an experimental factor.

Main effect: The average change in a response associated with changing one factor across its levels.

Interaction effect: A condition in which the effect of one factor depends on the level of another factor.

Full factorial design: An experiment containing every combination of the selected factor levels.

Fractional factorial design: A carefully chosen subset of a full factorial design.

Randomization: The use of random run order to reduce bias from time or unknown influences.

Blocking: Grouping similar experimental conditions to isolate nuisance variation.

Replication: Repetition of an experimental treatment to estimate experimental variation.

ANOVA: A statistical method for partitioning response variation among model effects and error.

Response surface: A model or graphical representation of response behavior across factor settings.

Robustness: The ability of a process to perform acceptably despite ordinary noise or variation.

FMEA: A systematic method for identifying, evaluating, and controlling potential failure modes.

Desirability function: A method for combining multiple response objectives into an optimization criterion.

Review Questions

Short Answer Questions

1. Why should a proposed improvement be linked to a verified cause?

2. Distinguish a pilot study from an informal trial.

3. What is an interaction effect in DOE?

4. Why are randomization and blocking used in experiments?

5. What information should be transferred from Improve to Control?

Long Answer Questions

1. Explain how creativity, benchmarking, decision matrices, and FMEA can be integrated to select an improvement.

2. Describe the logic and managerial value of a two-level full factorial design.

3. Discuss the limitations of one-factor-at-a-time experimentation.

4. Explain how response analysis supports optimization and robust operating conditions.

5. Describe the major elements of an implementation and scale-up plan.

Analytical / Application Questions

1. A process team finds that a new setting lowers the average defect rate but increases variation during night shifts. How should the team investigate and decide whether to implement the change?

2. Design a small factorial experiment for a service process in which response time may depend on staffing level, digital self-service, and case complexity. Identify the factors, levels, response, and possible interaction.

3. A pilot produces favorable results, but the intervention introduces a data-integrity risk. Propose a risk-control and governance plan before full deployment.

22. Multiple Choice Questions with Answers

Question 1

Which statement best describes the purpose of the Improve phase in DMAIC?

A. To define the project charter and customer requirements

B. To convert verified causes into tested and sustainable countermeasures

C. To establish baseline measurement-system capability only

D. To monitor long-term process stability after implementation

Correct Answer: B. To convert verified causes into tested and sustainable countermeasures

Question 2

A team changes factor A and observes its effect only when factor B is at its low level. Which concept is most relevant?

A. Blocking

B. Replication

C. Interaction effect

D. Measurement resolution

Correct Answer: C. Interaction effect

Question 3

Why is one-factor-at-a-time experimentation often inadequate?

A. It always requires randomization

B. It cannot identify interactions efficiently

C. It prevents measurement of main effects

D. It can only be used for service processes

Correct Answer: B. It cannot identify interactions efficiently

Question 4

What is the principal purpose of a pilot study?

A. To guarantee permanent process control

B. To replace all statistical analysis

C. To evaluate a proposed change under limited, controlled, and representative conditions

D. To eliminate the need for employee involvement

Correct Answer: C. To evaluate a proposed change under limited, controlled, and representative conditions

Question 5

A fractional factorial design is especially useful when a team wants to:

A. Screen many factors with fewer runs

B. Study only one factor at a time

C. Avoid all possible aliasing

D. Guarantee that higher-order interactions are large

Correct Answer: A. Screen many factors with fewer runs

Question 6

In an experiment, the response is best understood as:

A. A controllable input variable

B. A specific setting of a factor

C. The measured output or performance characteristic

D. The order in which runs are conducted

Correct Answer: C. The measured output or performance characteristic

Question 7

Why might a slightly inferior average setting be selected over the statistically best setting?

A. It may be more robust to normal process noise

B. It eliminates the need for a control plan

C. It always has a lower implementation cost

D. It makes interactions irrelevant

Correct Answer: A. It may be more robust to normal process noise

Question 8

Which practice most directly protects an experiment from time-related bias?

A. Increasing the number of response measures without planning

B. Randomizing the run order

C. Selecting only favorable observations

D. Using a single operator for every run

Correct Answer: B. Randomizing the run order

Question 9

In the illustrative laboratory case, why did barcode use and electronic calculation have greater combined value than either alone?

A. They reduced the need for any validation

B. They demonstrated an interaction between the interventions

C. They increased batch-size variation deliberately

D. They made the response impossible to measure

Correct Answer: B. They demonstrated an interaction between the interventions

Question 10

Which statement reflects sound professional governance during Improve?

A. Unfavorable pilot results should be omitted if the solution is strategically important

B. Statistical significance alone determines implementation

C. Digital changes should be validated and governed when they affect data integrity

D. A successful pilot never requires Control-phase monitoring

Correct Answer: C. Digital changes should be validated and governed when they affect data integrity