12. Experimental Design and Improvement Optimization

Introduction and Context

Quality improvement is often described as a sequence of actions: identify a problem, determine its causes, implement a solution, and verify the result. In complex processes, however, improvement cannot reliably be achieved through intuition, isolated trials, or changing one factor at a time. Processes are affected by several variables simultaneously, and the effect of one variable may depend on the level of another. Experimental design provides a disciplined way to learn how a process behaves and how it can be improved with minimum waste of time, material, and managerial attention.

Design of experiments (DOE) is a structured methodology for planning, conducting, analysing, and interpreting controlled changes to process inputs. Its purpose is not merely to discover statistical significance. A successful experiment converts process knowledge into a practical decision: which settings should be used, under what conditions, with what level of confidence, and at what cost or risk. In total quality management, DOE therefore links statistical analysis with customer requirements, process capability, operational control, and continual improvement.

The chapter develops comparative experiments, factorial designs, interaction effects, response-surface methods, and robust design. It also considers randomization, replication, blocking, practical significance, ethical conduct, and the use of digital analytics. These ideas extend the logic of earlier quality tools such as process mapping, cause-and-effect analysis, measurement-system analysis, and statistical process control. They are especially valuable when observational data or brainstorming have narrowed the problem but have not established causation.

Learning Objectives

After studying this chapter, the learner should be able to:

1. Distinguish observational studies, comparative experiments, and designed experiments.

2. Identify responses, factors, levels, treatments, nuisance variables, and experimental units.

3. Plan experiments using randomization, replication, blocking, and appropriate controls.

4. Construct and interpret full and fractional factorial designs.

5. Explain main effects, interaction effects, confounding, and aliasing.

6. Use response-surface methods to locate and optimize process conditions.

7. Apply robust-design principles to reduce sensitivity to noise factors.

8. Separate statistical significance from practical and economic significance.

9. Evaluate experimental results and translate them into sustainable process improvements.

From Process Questions to Experimental Questions

A process problem should be expressed as a question that an experiment can answer. “Why are defects increasing?” is too broad. A more useful question might be: “How do curing temperature, mixing time, and material grade affect tensile strength and defect rate, and is there a combination that meets the customer specification consistently?” The second formulation identifies measurable responses and controllable factors.

The response is the output measured from each experimental unit, such as cycle time, strength, dimensional error, yield, complaints, or energy consumption. A factor is an input deliberately varied by the investigator. A factor may be quantitative, such as pressure or dosage, or qualitative, such as supplier, machine, or operator method. A level is a particular value or category of a factor. A treatment is a specified combination of factor levels. The experimental unit is the smallest unit independently exposed to a treatment, such as a batch, component, transaction, patient, or production run.

The noise factors are variables that influence the response but are not the principal objects of study. They may be controlled, measured, blocked, or deliberately varied to assess robustness. A confounding variable is associated with both the treatment and the response in a way that prevents their effects from being separated. For example, if one machine always receives the new method and another always receives the old method, machine and method are confounded.

A strong experimental question specifies the population or process, the response measure, the factor ranges, the comparison of interest, the acceptable risk of error, and the decision rule. It also defines what would count as a worthwhile improvement. A statistically detectable reduction in defects may be operationally irrelevant if it does not offset implementation cost or if it creates unacceptable variation elsewhere.

Comparative Experiments and the Logic of Causation

The simplest comparative experiment compares two treatments. Suppose a service centre tests two scripts for reducing average customer-handling time. Customers or service agents are assigned to the scripts according to a planned design, and the resulting times are compared. The difference in average response estimates the treatment effect, provided that allocation is independent of other influences.

Randomization is central because it balances known and unknown nuisance variables, at least in expectation. It protects the comparison from systematic bias. If the most experienced agents select the new script, an observed improvement may reflect experience rather than the script. Random assignment of experimental units, or random assignment within suitable groups, reduces this risk.

Replication means applying each treatment to multiple independent experimental units. It permits estimation of experimental error and improves the precision of treatment comparisons. Repeating a measurement on the same unit is not automatically replication; it may be repeated measurement. If ten readings are taken from one batch, the batch may still represent only one independent experimental unit. Treating correlated readings as independent produces pseudo-replication and overly optimistic conclusions.

A control or reference treatment provides a meaningful benchmark. It may be the current standard, a placebo, a baseline condition, or a target process. Controls are particularly important where natural variation or time trends could otherwise be mistaken for treatment effects. When the process changes over time, experimental runs should be interspersed rather than conducted in treatment order.

Blocking is used when a known nuisance source is expected to create variation. Units are grouped into relatively homogeneous blocks, and treatments are randomized within each block. Examples include blocking by day, raw-material lot, operator, machine, hospital ward, or environmental condition. Blocking does not eliminate the nuisance effect; it prevents that effect from obscuring the comparison. A randomized complete block design is often more informative than completely randomizing all units when block-to-block variation is substantial.

An experiment should be randomized whenever feasible, replicated sufficiently for the intended precision, blocked when important heterogeneity is known, and conducted under a written protocol. The protocol should specify measurement methods, treatment settings, order of runs, handling of deviations, safety requirements, and criteria for stopping or repeating the study.

Factorial Thinking: Main Effects and Interactions

Changing one factor at a time is attractive because it appears simple. It is inefficient, however, and can miss interactions. A factorial experiment varies two or more factors in combinations. In a two-level, two-factor design, factor A and factor B are each tested at a low and high level, producing four treatment combinations: low-low, high-low, low-high, and high-high.

The main effect of A is the change in the average response when A moves from its low to high level, averaged over the levels of B. The main effect of B is defined similarly. An interaction occurs when the effect of one factor changes according to the level of another. On an interaction plot, parallel lines suggest little interaction, whereas nonparallel or crossing lines indicate interaction.

For example, suppose a coating process is tested at two spray pressures and two curing temperatures. Higher pressure may improve surface coverage at low temperature but reduce quality at high temperature because excess material traps solvent. The best pressure cannot therefore be selected without considering temperature. A recommendation based only on the average pressure effect could be wrong for a particular operating condition.

In a two-level factorial design, the estimated interaction is based on the contrast between the changes caused by A at the two levels of B. Analysis of variance (ANOVA) partitions total variation into components associated with factors, interactions, and error. A small p-value indicates that the observed effect would be relatively unlikely under a no-effect model, subject to the assumptions and design. It does not establish practical importance, causation without sound design, or universal applicability beyond the studied region.

A full factorial design examines every combination of selected levels. It is highly informative but can become large: a design with k factors at two levels requires 2^k treatment combinations. Fractional factorial designs use a carefully selected fraction of those combinations. They reduce runs but create aliasing, in which certain effects cannot be estimated separately. For instance, a main effect may be aliased with a two-factor interaction. Resolution describes the severity of such confounding; higher-resolution designs generally provide clearer estimates of important lower-order effects.

Screening designs, such as Plackett–Burman designs or other economical fractional designs, are useful when many candidate factors exist and only a few are expected to be influential. Screening should not be confused with final optimization. Once important factors have been identified, additional experiments are usually required to estimate curvature, interactions, and precise operating settings.

Response-Surface Methods and Improvement Optimization

When a process response may curve across the operating region, two-level experiments are insufficient. Response-surface methodology (RSM) models the response as a function of quantitative factors. A commonly used second-order model is:

Y = β0 + ΣβiXi + ΣβiiXi² + ΣΣβijXiXj + ε

where Y is the response, Xi are coded factors, β coefficients describe linear, quadratic, and interaction effects, and ε represents unexplained variation. The quadratic terms allow the model to represent a maximum, minimum, or saddle-shaped surface.

Central composite designs and Box–Behnken designs are common RSM designs. They use combinations of factorial or fractional points, centre points, and additional points that permit estimation of curvature. Centre points also provide a check for process drift and pure error, although they cannot replace replication across the design region.

Optimization begins with a fitted model but does not end with it. Residual analysis should examine normality assumptions where relevant, constant variance, independence, outliers, and model adequacy. A contour plot or three-dimensional surface can help communicate the relationship, but visual interpretation should be supported by numerical prediction intervals and confirmation runs.

Many quality problems involve several responses. Increasing speed may reduce cost but increase defects; raising strength may increase energy consumption. Desirability functions transform each response into a scale from unacceptable to fully desirable, and a combined desirability measure can identify a compromise. Management must still assign priorities and constraints. Mathematical optimization cannot decide whether a small quality gain justifies a large environmental or maintenance burden.

A confirmation experiment is essential. The predicted optimum should be run under realistic production conditions, preferably with independent material lots, operators, or time periods. The result should be compared with the prediction and with the current process. If the optimum lies at a boundary, the experiment may indicate that the true optimum is outside the studied region; it may also signal that safety, capability, or cost constraints have been overlooked.

Robust Design and Taguchi Thinking

Optimization of the average response is not sufficient when performance varies with environment or usage. Robust design seeks settings that are relatively insensitive to noise factors. In Taguchi’s framework, control factors are selected by the organization, while noise factors represent conditions that are difficult or costly to control. An orthogonal array can study combinations efficiently, and a signal-to-noise ratio summarizes performance according to the objective, such as “smaller is better,” “larger is better,” or “nominal is best.”

The underlying quality principle is broadly useful: design the process so that ordinary variation has little effect on the customer-critical response. Robustness can be improved by selecting materials, tolerances, settings, or designs that reduce sensitivity, rather than by relying only on inspection or adjustment after failure.

Taguchi methods are valuable for planning economical studies, but their use should be critically informed. Signal-to-noise transformations can obscure the original response scale, and simplified analyses may overlook important interactions or uncertainty. Modern robust-design studies often use explicit response models, variance modelling, mixed-effects methods, or combined control-noise experiments. The choice should reflect the scientific question and the structure of the data.

Practical Significance, Uncertainty, and Decision Quality

Statistical significance and practical significance answer different questions. Statistical significance concerns evidence against a specified null hypothesis. Practical significance concerns whether the size of the effect matters to customers, capability, cost, safety, sustainability, or strategy. A very large sample may identify a trivial difference as statistically significant, while a meaningful improvement may fail to achieve significance in a small, noisy study.

Decision-makers should examine estimated effects, confidence intervals, prediction intervals, process capability, and economic consequences. A confidence interval that excludes zero may still include effects too small to justify implementation. Conversely, an interval that includes a modest beneficial effect may warrant further experimentation if the potential gain is substantial and the intervention is low risk.

Power analysis helps determine the number of experimental units needed to detect a specified effect with a chosen probability, given an estimate of variation and an acceptable Type II error. Underpowered studies waste resources and produce ambiguous results; excessively large studies may be unnecessarily expensive. Sample-size planning should account for clustering, blocking, repeated measures, attrition, and multiple comparisons.

Multiple testing also requires care. Examining many responses, factors, and subgroups increases the chance of finding an apparently favourable result by chance. Predefined primary responses, hierarchical analysis, adjustment procedures, and transparent reporting reduce selective interpretation. Data transformation, removal of inconvenient observations, or repeated experimentation until a preferred p-value appears are unacceptable practices unless justified and documented.

A Structured DOE Workflow

A practical DOE project can proceed through the following connected stages:

1. Define the improvement opportunity. Translate the business or customer problem into measurable responses and a clear decision objective.

2. Establish the measurement system. Confirm operational definitions, calibration, resolution, repeatability, reproducibility, and data integrity.

3. Map the process and select factors. Use process knowledge, historical data, risk analysis, and subject-matter expertise to identify controllable and noise factors.

4. Specify the experimental region. Select safe, feasible levels that cover a meaningful operating range without extrapolation beyond evidence.

5. Choose the design. Select comparative, factorial, fractional, blocked, split-plot, response-surface, or robust design according to the question and restrictions.

6. Plan randomization and replication. Decide the experimental unit, run order, number of replicates, blocks, and treatment of missing data.

7. Conduct and monitor the experiment. Record actual settings, deviations, environmental conditions, and unusual events; do not silently discard runs.

8. Analyse the data. Use graphical analysis, effect estimates, ANOVA or regression, residual diagnostics, and uncertainty intervals.

9. Interpret technically and economically. Consider practical effect size, capability, cost, risk, sustainability, and implementation constraints.

10. Confirm and control. Validate the selected settings in realistic conditions, update standard work and control plans, and monitor performance after implementation.

Illustrative Case: Reducing Seal Defects in a Medical-Device Packaging Process

Background

The following is an illustrative case. A medical-device manufacturer observes that pouches occasionally fail a pressure-leak test. The defect rate is below the internal escalation threshold but above the customer target. Engineers suspect sealing temperature, dwell time, and jaw pressure. Production also runs on three machines and uses several material lots.

Situation and Problem

Early trials changed temperature alone and produced inconsistent results. The trials were performed mainly on one machine during a single shift, so machine and shift effects could not be separated. Management wants to reduce defects without increasing seal width, cycle time, or material waste.

Analysis and Design

The team first verifies the leak-test measurement system and defines the response as the proportion of failed pouches, with seal strength as a secondary response. A two-level factorial experiment is selected for temperature, dwell time, and pressure. Because machine-to-machine variation is expected, machine is used as a block. Treatment order is randomized within each machine, and material lot is recorded as a potential noise factor. Replication is planned across production runs rather than by repeatedly testing the same pouch.

The analysis shows that temperature has a strong main effect, but its benefit depends on dwell time. Pressure has a smaller average effect but interacts with temperature. A model containing the important main effects and interactions explains substantially more variation than a main-effects-only model. Residual plots reveal no serious variance pattern, but one run has an incorrectly recorded pressure setting; it is investigated and retained with a documented correction rather than silently deleted.

Approach and Solution

Because the best combination appears near the high-temperature boundary, the team conducts a response-surface follow-up using temperature and dwell time while holding pressure in a feasible range. The objective is to minimize leak defects while maintaining seal strength and cycle-time limits. A confirmation run uses all three machines and two independent material lots. The selected setting is not the point with the lowest predicted defect rate alone; it is a slightly less aggressive setting with a wider operating window and lower sensitivity to material-lot variation.

Outcome

The confirmation results meet the defect target and remain within the seal-strength specification. The organization updates standard work, machine-setting verification, preventive-maintenance checks, and the process control plan. Performance is monitored using defect-rate data and periodic capability analysis. The team also records the design region and assumptions so that future material changes do not automatically inherit the old setting.

Learning Points

The case illustrates why one-factor-at-a-time trials can miss interactions, why blocking protects comparisons from machine variation, and why confirmation must reflect production conditions. It also demonstrates that the statistically best predicted point is not always the best managerial choice. Robustness, operating margin, cost, safety, and maintainability influence the final decision.

Challenges, Risks, and Common Mistakes

DOE fails when the problem is poorly framed, the measurement system is incapable, or factor levels are selected without process knowledge. Extreme settings may create unsafe or irrelevant conditions, whereas narrow settings may conceal important behaviour. Noncompliance with the randomization plan, treatment contamination, operator learning, time trends, and equipment drift can bias results.

Other common mistakes include treating repeated measurements as independent replicates, ignoring missing runs, fitting an overly complex model to limited data, interpreting a main effect in the presence of a strong interaction, and optimizing within a region that was never adequately studied. Statistical software can produce precise-looking output from an invalid design; software does not repair confounding or poor measurement.

Organizational resistance is another challenge. Operators may fear that experiments threaten performance evaluations, while managers may demand immediate conclusions before adequate replication. Leaders should protect the integrity of the experiment, communicate its purpose, and distinguish learning trials from routine production targets.

Governance, Ethics, and Digital Practice

Experiments involving people, health, safety, personal data, or regulated products require appropriate review, consent, privacy safeguards, traceability, and compliance with applicable standards. In manufacturing, no experiment should knowingly expose customers or workers to unacceptable risk. Changes affecting validated processes must follow change-control requirements. Data provenance, access control, versioned analysis files, and retained raw data support auditability.

Digital manufacturing systems, laboratory information systems, and statistical platforms can automate run-order generation, sensor capture, model fitting, and control-plan updates. Machine learning may help identify candidate factors or detect nonlinear patterns, but predictive association does not replace designed experimentation when causal decisions are required. Automated optimization should include constraints, human review, drift detection, and safeguards against optimizing a proxy while damaging an unmeasured quality characteristic.

Current Developments and Best Practice

Modern experimentation increasingly combines physical tests with digital twins, simulation, historical process data, and sequential adaptive designs. Bayesian methods can incorporate prior engineering knowledge and update predictions as evidence accumulates. Optimal-design algorithms can select informative runs when experiments are expensive, while sequential experimentation allows early learning to guide later trials. These approaches are particularly useful in high-mix production, bioprocessing, and complex service systems.

Nevertheless, sophisticated methods should not displace disciplined fundamentals. Effective practice includes:

• defining the decision before collecting data;
• involving process owners, statisticians, operators, quality professionals, and safety specialists;
• validating the measurement system;
• preserving randomization and treatment identity;
• using replication and blocking deliberately;
• modelling interactions and curvature when justified;
• reporting estimates and uncertainty, not only p-values;
• conducting confirmation runs under realistic conditions; and
• embedding the improvement in standard work, monitoring, and governance.

Chapter Summary

Experimental design provides a causal and economical framework for improving processes. Comparative experiments depend on appropriate controls, randomization, replication, and clearly defined experimental units. Blocking handles known nuisance variation, while factorial designs reveal both main effects and interactions. Fractional factorial designs support efficient screening but require attention to aliasing and resolution. Response-surface methods model curvature and help locate feasible operating regions, whereas robust design seeks settings that remain effective despite noise factors.

Sound DOE joins statistical evidence with engineering judgement and managerial evaluation. Statistical significance must be considered alongside effect size, uncertainty, capability, cost, safety, sustainability, and implementation risk. A complete improvement cycle includes measurement validation, design planning, execution, analysis, confirmation, and process control. Digital and advanced analytical tools can strengthen this work, but they cannot substitute for valid experimental logic and ethical governance.

Key Terms

Analysis of variance (ANOVA): A method for partitioning response variation into components attributable to model effects and error.

Blocking: Grouping experimental units by a known nuisance source and randomizing treatments within groups.

Confounding: A condition in which effects of two or more sources cannot be separated by the design.

Control factor: A process variable whose setting can be selected or adjusted by the organization.

Design of experiments (DOE): A systematic method for planning, conducting, and analysing controlled process changes.

Desirability function: A method for combining multiple responses into a common scale for optimization.

Effect: The estimated change in a response associated with a factor or treatment.

Experimental unit: The smallest independently treated unit in an experiment.

Factor: An input deliberately varied in a designed study.

Factorial design: A design that tests combinations of levels of two or more factors.

Interaction: A condition in which the effect of one factor depends on another factor’s level.

Level: A particular value or category assigned to a factor.

Main effect: The average effect of changing one factor across levels of other factors.

Noise factor: A variable that affects performance but is difficult or undesirable to control routinely.

Power: The probability of detecting a specified effect when that effect truly exists.

Randomization: Allocation or ordering by chance to protect comparisons from systematic bias.

Replication: Applying treatments to multiple independent experimental units.

Response surface methodology: Modelling and optimizing a response as a function of quantitative factors.

Robust design: Designing settings or products to reduce sensitivity to noise factors.

Statistical significance: Evidence that observed data are inconsistent with a specified no-effect hypothesis, under model assumptions.

Review Questions

Short Answer Questions

1. What is the difference between a factor, a level, and a treatment?

2. Why is randomization important in a comparative experiment?

3. Distinguish replication from repeated measurement.

4. What does an interaction effect mean?

5. Why is a confirmation run required after response-surface optimization?

Long Answer Questions

1. Explain how randomization, replication, blocking, and control treatments jointly improve the validity of a designed experiment.

2. Compare full factorial, fractional factorial, and response-surface designs, including their appropriate uses and limitations.

3. Discuss the difference between statistical significance and practical significance in quality improvement decisions.

4. Explain robust design and evaluate the usefulness and limitations of signal-to-noise approaches.

5. Develop a complete DOE workflow for improving a process with multiple quality responses and operational constraints.

Analytical / Application Questions

1. A process team changes temperature and pressure one at a time and concludes that pressure has no effect. Explain why this conclusion may be invalid and propose a suitable factorial design.

2. An experiment produces a statistically significant 0.2% defect reduction but increases energy cost by 8%. How should management evaluate the result before implementation?

3. Three machines produce noticeably different baseline outputs, and four process settings must be compared. Design an experiment that addresses machine variation and explain how its results should be interpreted.

22. Multiple Choice Questions with Answers

Question 1

Which feature most directly protects a treatment comparison from systematic differences among experimental units?

A. Randomization

B. Increasing the number of response variables

C. Using only centre points

D. Fitting a quadratic model

Correct Answer: A. Randomization

Question 2

If the effect of curing temperature changes according to the level of pressure, the process exhibits:

A. Replication

B. An interaction

C. Blocking

D. Aliasing only

Correct Answer: B. An interaction

Question 3

What is the primary purpose of blocking by machine in a production experiment?

A. To eliminate the machine effect entirely

B. To ensure every machine receives only one treatment

C. To reduce the obscuring influence of machine-to-machine variation

D. To replace randomization

Correct Answer: C. To reduce the obscuring influence of machine-to-machine variation

Question 4

A fractional factorial design is especially useful when:

A. Every possible treatment combination must be tested

B. Many factors must be screened with limited runs

C. Only one response can be measured

D. No factor levels can be specified

Correct Answer: B. Many factors must be screened with limited runs

Question 5

Which result most strongly suggests curvature in a two-level experiment supplemented with centre points?

A. The treatment labels are qualitative

B. The centre-point mean differs systematically from the average predicted by the linear model

C. The response has no measurement error

D. All factor effects are exactly zero

Correct Answer: B. The centre-point mean differs systematically from the average predicted by the linear model

Question 6

Why can repeated measurements on one batch fail to provide true replication?

A. They may not be statistically independent experimental units

B. They always increase confounding

C. They prevent any response measurement

D. They automatically create a fractional design

Correct Answer: A. They may not be statistically independent experimental units

Question 7

Which statement best distinguishes practical significance from statistical significance?

A. Practical significance is determined only by the p-value

B. Statistical significance always implies economic benefit

C. Practical significance considers whether the effect matters operationally, economically, or to customers

D. The two terms have identical meanings

Correct Answer: C. Practical significance considers whether the effect matters operationally, economically, or to customers

Question 8

In robust design, noise factors are primarily studied to determine:

A. How to make performance less sensitive to difficult-to-control conditions

B. Which operators should be excluded

C. Whether randomization is unnecessary

D. How to increase measurement resolution only

Correct Answer: A. How to make performance less sensitive to difficult-to-control conditions

Question 9

A model predicts an optimum at the edge of the tested region. What is the most appropriate next action?

A. Assume the global optimum has been found

B. Ignore the boundary and report the model without qualification

C. Conduct further experimentation in a justified direction and perform confirmation

D. Remove all quadratic terms

Correct Answer: C. Conduct further experimentation in a justified direction and perform confirmation

Question 10

Which practice is most appropriate when analysing many responses and treatment comparisons?

A. Report only the most favourable p-value

B. Define primary responses in advance and interpret multiplicity transparently

C. Delete results that do not support the hypothesis

D. Stop the experiment whenever any result is significant

Correct Answer: B. Define primary responses in advance and interpret multiplicity transparently