NG BB 47 Basic Design of Experiments

UNCLASSIFIED / FOUO

UNCLASSIFIED / FOUO

National Guard
Black Belt Training

Module 47

Basic Design of
Experiments (DOE)

UNCLASSIFIED / FOUO

UNCLASSIFIED / FOUO

UNCLASSIFIED / FOUO

CPI Roadmap – Improve
8-STEP PROCESS
6. See
1.Validate 2. Identify 3. Set 4. Determine 5. Develop 7. Confirm 8. Standardize
Counter-
the Performance Improvement Root Counter- Results Successful
Measures
Problem Gaps Targets Cause Measures & Process Processes
Through

Define Measure Analyze Improve Control

TOOLS
ACTIVITIES
•Brainstorming
• Develop Potential Solutions
•Replenishment Pull/Kanban
• Develop Evaluation Criteria
•Stocking Strategy
• Select Best Solutions
•Process Flow Improvement
• Develop Future State Process Map(s)
•Process Balancing
• Develop Pilot Plan
•Standard Work
• Pilot Solution
•Quick Change Over
• Develop Full Scale Action/
•Design of Experiments (DOE)
Implementation Plan
•Solution Selection Matrix
• Complete Improve Gate
•‘To-Be’ Process Mapping
•Poka-Yoke
•6S Visual Mgt
•RIE

Note: Activities and tools vary by project. Lists provided here are not necessarily all-inclusive. UNCLASSIFIED / FOUO 2

UNCLASSIFIED / FOUO

Learning Objectives
 Learn benefits of DOE methodology
 Discuss differences between DOE and trial and
error (one-factor-at-a-time) approaches to
experimentation
 Learn basic DOE terminology
 Distinguish between the concepts of full
and fractional factorial designs
 Use Minitab to run and analyze a DOE
 Use results of DOE to drive statistically
significant improvements

Basic Design of Experiments UNCLASSIFIED / FOUO 3

UNCLASSIFIED / FOUO

Helicopter Simulation
Phase One

UNCLASSIFIED / FOUO 4

UNCLASSIFIED / FOUO

Exercise: Helicopter Simulation
 Customers at CHI (Cellulose Helicopters Inc.) have
been complaining about the limited flight time of CHI
helicopters
 Management wants to increase flight time to improve
customer satisfaction
 You are put in charge of this improvement project


UNCLASSIFIED / FOUO

Exercise: Constraints
 Project Mission: Find the combination of factors that
maximize flight time
 Project Constraints:
 Budget for testing = $1.5 M
 Cost to build one prototype = $100,000
 Cost per flight test = $10,000
 Prototype once tested can not be altered
 See allowable flight test factors and parameters on the
next page


UNCLASSIFIED / FOUO

Exercise: Test Factors and Parameters
 Paper Type Regular Card stock
 Paper Clip No Yes
 Taped Body No 3 in of tape
 Taped Wing Joint No Yes
 Body Width 1.42 in 2.00 in
 Body Length 3.00 in 4.75 in
 Wing Length 3.00 in 4.75 in


UNCLASSIFIED / FOUO

Exercise: Roles & Responsibilities
 Lead Engineer – Leads the team and makes final
decision on which prototypes to build and test
 Test Engineer – Leads the team in conducting the test
and has final say on how test are conducted
 Assembly Engineer – Leads the team in building
prototypes and has final say on building issues
 Finance Engineer – Leads the team in tracking
expenses and keeping the team on budget
 Recorder – Leads the team in recording data from the
trials

UNCLASSIFIED / FOUO

Exercise: Phase One Deliverables
 Prepare a Phase One Report showing:
 Recommendation for optimal design
 Predicted flight time at optimal setting
 How much money was spent
 Description of experimental strategy used
 Description of analysis techniques used
 Recommendations for future tests


UNCLASSIFIED / FOUO

Introduction to DOE


UNCLASSIFIED / FOUO

How Do We Learn?
1. Significant Event 2. Somebody Sees It

3. Research How can we learn
more efficiently?


UNCLASSIFIED / FOUO

How Do We Learn? (Cont.)
 Products and processes are continually providing data
that could lead to their improvement - so what has
been missing? There are several possibilities:
 We are not collecting and analyzing the data provided
 We are not proactive in data collection
 We are unable to translate the data into information
 A significant event has not occurred
“In order to learn, two things must occur simultaneously:
something must happen (informative event) and someone
must see it happen (perceptive observer).” – George Box


UNCLASSIFIED / FOUO

How Do We Improve?
 By creating significant events and observing them, we
can obtain knowledge faster
 That is basically what occurs in a designed
experiment
 Let‟s look at an example of these two things occurring
(significant event and perceptive observer)
simultaneously


UNCLASSIFIED / FOUO

Champagne Example
 Wine – The fermented juice of
fresh grapes used as a beverage.
Wine has been in existence since
the beginning of recorded history
 Champagne – A clear, sparkling
liquid made by way of the second
fermentation of wine. First
discovered by a French monk in
the late 1600s


UNCLASSIFIED / FOUO

Need Improved Observation
 Need to make sure that naturally occurring informative
events are brought to the attention of the perceptive
observer!
 Improved observation increases the probability of observing
naturally occurring informative events so appropriate action
can be taken.

SPC tools and techniques improve observation, but we
must wait for an event to happen in order to observe it

UNCLASSIFIED / FOUO

Passive Observation Is Not Enough
 We need to induce occurrence of informative events.
 An experiment is set-up so that an informative event
will occur!
Designed Experimentation – The manipulation of
controllable factors (independent variables) at different levels
to see their effect on some response (dependent variable)

 By manipulating inputs to see how the output
changes, we can understand and model Y (a
dependent variable) as a function of X (an
independent variable).


UNCLASSIFIED / FOUO

What Is Experimental Design?

Inputs (Factors) Outputs (Responses)

People
Responses related to
Material
producing a product

Equipment Experimental Process:
A controlled blending Responses related to
Policies
of inputs which completing a task

Procedures generates corresponding
measurable outputs. Responses related to
Methods
performing a service
Environment

From Understanding Industrial Designed Experiments, Schmidt & Launsby

UNCLASSIFIED / FOUO

Example of a Recruiting Process
Inputs Outputs
(Factors) (Responses)

Job Description

Marketing

Candidate Pool

Economic Environment Process:
Recruiting
Type of Job

Location of Job Hire Quickly

Job Application Process Hire Best Candidate

EEO Requirements Hire at Competitive Pay

DOE was originally used for manufacturing quality applications - it has now
expanded to many other areas where performance characteristics are of interest

UNCLASSIFIED / FOUO

Methods of Experimentation
 Experimentation has been used for a long time.
Some experiments have been good, some not so
good
 Our early experiments can be grouped into the
following general categories:
1. Trial and Error
2. One-Factor-at-a-Time (OFAT)
3. Full Factorial
4. Fractional Factorial
5. Others


UNCLASSIFIED / FOUO

Trial and Error - Increase Gas Mileage
 Problem: Gas mileage for car is 20 mpg. Would like to get > 30
mpg.
 Factors:
 Change brand of gas
 Change octane rating
 Drive slower
 Tune-up car
 Wash and wax car
 New tires
 Change tire pressure
 Remove hood ornament and external radio antenna


UNCLASSIFIED / FOUO

One-Factor-at-a-Time (OFAT)- Gas Mileage
 Problem: Gas mileage for vehicle is 20 mpg. Would like to get > 30 mpg
MPG Results

Speed Octane Tire Pressure Miles per Gallon
55 85 30 25
65 85 30 23
55 91 30 27
55 85 35 27

 How many more runs would you need to figure out the best configuration of
variables?
 How can you explain the above results?
 If there were more variables, how long would it take to get a good solution?
 What if there‟s a specific combination of two or more variables that leads to the
best mileage (the optimum)?


UNCLASSIFIED / FOUO

Results
Miles per Gallon as a Function of Speed and Tire Pressure

75
17
70
 How would we find
65
Speed (mph)

26 this optimum with
60
OFAT testing?
55 18 23 26 26 20
50
26  How would we
45
know that we‟d
18 found it?
40
35 35
30 Optimum MPG

26 28 30 32 34 36 38
Tire Pressure (lbs.)

UNCLASSIFIED / FOUO

One-Factor-at-a-Time (OFAT)
 While OFAT is simple, it is inefficient in determining
optimal results:
 Unnecessary experiments may be run
 Time to find causal factors is significant
 Don‟t know the effects of changing one factor while
other factors are also changing (no model)
 Inability to detect or learn about how factors work
together to drive the response

Is there a better way?


UNCLASSIFIED / FOUO

Cake Example - Interactions
 An Interaction occurs when the
effect of one factor, X1, on the
response, Y, depends on the setting
(level) of another factor, X2:
Y = f(x)

 For example, when baking a cake, the
temperature that you set the oven at
is dependent on the time that the cake
will be in the oven.


UNCLASSIFIED / FOUO

 Where would you set Time to get a good cake?
 How would you experiment on this process
to learn about this interaction?
Temp = 100 degrees Temp = 500 degrees
Time = 45 minutes Time = 20 minutes
Time

Temp = 100 degrees Temp = 500 degrees
Time = 20 minutes Time = 45 minutes

100 Temp 500


UNCLASSIFIED / FOUO

 Duncan Hines used designed
experiments in the 50‟s on their cake
mixes.
 Their goal was a robust design for
the most consistent product.


UNCLASSIFIED / FOUO

Why Use DOE?
“Often we have used a trial and error
approach to testing, or just changed one
variable at a time. Why is a statistically
designed experiment better?”
 The structured methodology provides a directed approach to avoid time wasted
with “hunt and peck” - don‟t need 30 years of experience to design the tests
 The designed experiment gives a mathematical model relating the variables and
responses - no more experiments where you can‟t draw conclusions
 The model is easily optimized, so you know when you‟re done
 The statistical significance of the results is known, so there is much greater
confidence in the results
 Can determine how multiple input variables interact to affect results


UNCLASSIFIED / FOUO

Full Factorial DOE
 Full Factorial examines every possible
combination of factors at the levels tested. The full
factorial design is an experimental strategy that
allows us to answer most questions completely.
 Full factorial enables us to:
 Determine the Main Effects that the factors being
manipulated have on the response variable(s)
 Determine the effects of factor interactions on the
response variables
 Estimate levels at which to set factors for best
results


UNCLASSIFIED / FOUO

Full Factorial
Minimum number of tests for a full factorial experiment: Xk
X = # of levels, k = # of factors

Factors
Level 2 3 4
2 4 8 16 # of Tests
3 9 27 81

Adding another level significantly increases the number of tests!

 Full Factorial Advantages
 Information about all effects
 Information about all interactions
What can we do when
 Quantify Y=f(x)
resources are limited?
 Limitations
 Amount of resources needed
 Amount of time needed


UNCLASSIFIED / FOUO

Full Factorial Notation
 2 level designs are the most common because
they provide a lot of information, but require the
fewest tests.
 The general notation for a full factorial design of 2
levels is:
2k = # Runs

 2 is the number of levels for each factor
(Range = High and Low)
 k is the number of factors to be investigated

 This is the minimum number of test runs required
for a full factorial

UNCLASSIFIED / FOUO

Full Factorial Experiment
Problem: Gas Mileage is 20 mpg

Speed Octane Tire Pressure Miles per Gallon
55 85 30 25
65 85 30 23
55 91 30 27
65 91 30 23
55 85 35 27
65 85 35 24
55 91 35 32
65 91 35 25
OFAT Runs
Do we think 32 is best?
 What conclusion do you make now?
 How many runs? MPG = f(Speed, Octane, Tire Pressure)
 How many runs at each level?


UNCLASSIFIED / FOUO

Fractional Factorial
 Looks at only a fraction of all the possible combinations contained in
a full factorial.
 If many factors are being investigated, information can be obtained
with smaller investment.
 Resources necessary to complete a fractional factorial are
manageable.
 Limitations - give up some interactions
 Benefits
 Economy
 Speed
 Fewer runs


UNCLASSIFIED / FOUO

Fractional Factorial Notation
 The general notation to designate a fractional factorial
design is:
k p
2R = # Runs

 2 is the number of levels for each factor
 k is the number of factors to be investigated
 2-p is the size of the fraction
(p = 1  1/2 fraction, p = 2  1/4 fraction, etc.)
 2k-p is the number of runs
 R is the resolution


UNCLASSIFIED / FOUO

Fractional Factorial Notation – Resolution
When we go to a fractional factorial design, we
are not able to estimate all of the interactions
The amount that we are able to estimate is
indicated by the resolution of an experiment
The higher the resolution, the more interactions
we can measure
 Example: The designation below means fifteen factors will be
investigated in 16 runs. This design is a resolution III:

1511
2 III Note: A deeper discussion of design resolution is beyond the scope of the lesson. The
content, above, is intended to only provide a brief explanation of the design resolution term.


UNCLASSIFIED / FOUO

Gas Mileage Example
 Problem: Gas mileage for vehicle is 20 mpg
Speed Octane Tire Pressure Mileage
(A) (B) (C) (Y)
55 85 35 27
65 85 30 23
55 91 30 27
65 91 35 25

 Compare with previous full factorial:
 How many runs?
 How much information?


UNCLASSIFIED / FOUO

DOE Will Help Us Identify Factors
 Factors which shift the average
A1 A2

 Longer line = greater effect
Main Effects Plot (data means) for Mileage

55 65 85 91 30 35
27.8

26.8

In the gas mileage
Mileage

25.8 example, Speed,
Octane, and Tire
24.8 Pressure all look to
have an effect on
23.8
average mileage.
Speed Octane Tire Pressure


UNCLASSIFIED / FOUO

 Factors which affect variation B2
B1
 Flat line = no effect
Main Effects Plot for Standard Deviation

55 65 85 91 30 35

3.0

2.5

Only Tire Pressure
Standard Dev

is affecting standard
2.0

1.5 deviation
1.0
Speed Octane Tire Pressure


UNCLASSIFIED / FOUO

 Factors which shift the average and C2
affect variation C1

Main Effects Plot (data means) for Mileage Main Effects Plot for Standard Dev
55 65 85 91 30 35 55 65 85 91 30 35
27.8 3.0

26.8 2.5

Standard Dev
Mileage

25.8 2.0

24.8 1.5

23.8 1.0
Speed Octane Tire Pressure Speed Octane Tire Pressure

Only Tire Pressure affects both the
average mileage and also the variability


UNCLASSIFIED / FOUO

 Factors which have no effect D1 = D2

Main Effects Plot (data means) for Mileage Main Effects Plot for Standard Dev
-1 1 -1 1 -1 1 -1 1
28
3.0

27
2.5

Standard Dev
Mileage

26
2.0

25
1.5

24
1.0
Driver Radio Driver Radio

An expanded study investigated the effect of driver and radio on
mileage. These factors show no effect. This is also valuable information,
because these factors can be set at their most economical (least cost) or
most convenient levels.

UNCLASSIFIED / FOUO

Benefits of DOE
 Determine input settings which optimize results and
minimize costs
 Quick screening for significant effects
 Obtain a mathematical model relating inputs and results
 Reduction in the number of tests required
 Verification of the statistical significance of results
 Identification of low-impact areas allows for increased
flexibility/tolerances
 Standardized methodology provides a directed approach


UNCLASSIFIED / FOUO

When Could I Use Design of Experiments?
 Identification of critical factors to improve
performance
 Identification of unimportant factors to reduce costs
 Reduction in cycle time
 Reduction of scrap/rework
 Scientific method for setting tolerances
 Whenever you see repetitive testing


UNCLASSIFIED / FOUO

DOE Review
 What does DOE offer us that trial and error
experimentation and OFAT do not?
 What are the differences between full and fractional
factorial DOE‟s?
 What is the minimum number of runs required for a
2-level, 3-factor full factorial experiment?


UNCLASSIFIED / FOUO

Minitab: Airline DOE Example
 A contract airline is interested in reducing overall late
take-off time in order to improve Soldier satisfaction
 Previous Black Belt work has identified 4 key process
input variables (KPIVs) that affect late time:
 Dollars spent on training
 Number of jets
 Number of employees
 % Overbooked


UNCLASSIFIED / FOUO

 Using the Airline DOE Data.mpj file and Minitab, the
instructor will walk the class through the following
activities:
 A DOE to identify which factors affect “minutes late” in
terms of both the mean and standard deviation
 Use the DOE results to determine new process settings
 Hypothesis test to prove statistical significance of change.


UNCLASSIFIED / FOUO

 Current settings for these factors are as follows:
 Dollars spent on training 200
 Number of jets 52
 Number of employees 850
 % Overbooked 15
 The target is zero minutes late, with a specification
of +/- 10 minutes.


UNCLASSIFIED / FOUO

 Our goal in this experiment is to reduce late take-
off times - we will measure late time in minutes
 Here are the factors and their levels that we are
going to investigate:
Factors Levels
 Dollars spent on training 100 300
 Number of Jets 50 55
 Number of Employees 800 900
 % Overbooked 0 25


UNCLASSIFIED / FOUO

 First, we need to set up the test matrix
 Select Stat>DOE>Factorial>Create Factorial
Design


UNCLASSIFIED / FOUO

 We left the default at: 2-level factorial design. That means we will
test each factor at 2 different levels
 We also selected 4 factors, since there are 4 variables that we want to
test in this experiment. Select Display Available Designs to display
possible experiments we can run…


UNCLASSIFIED / FOUO

 For 4 factors, we can either do an 8 run half-fraction or a
16 run full factorial. We will go with the 16 run full
factorial experiment.

 Click OK to return to the previous dialog box.

UNCLASSIFIED / FOUO

 Click the Designs button and highlight the 16-run Full
Factorial design.
 Leave the other settings at their defaults, click on OK.


UNCLASSIFIED / FOUO

 Next, click the Factors button
 Enter the names of the Factors – Change -1 and 1 to actual
levels per chart below

Factors
Levels
Dollars spent on training 100
300
Number of Jets 50 55
Number of Employees 800
900

UNCLASSIFIED / FOUO

 Click the Options button and uncheck Randomize runs.
 We do want to randomize our tests when we actually run an
experiment. However, for this in-class demo, it will be easiest if
everyone‟s screen is the same.


UNCLASSIFIED / FOUO

 Here are the tests that we need to run. Example, row 1 indicates that we
first need to collect data at the low level for all four factors.
(Tip: First check that you have the same test matrix. If you don‟t, it‟s
likely that you did not uncheck “Randomize Runs.”)


UNCLASSIFIED / FOUO

 Next, we collect the data.
 To allow us to measure variation, we need to run 3 repetitions at each
set of settings.
 Copy the data from the DOE data worksheet and paste into the design as
shown below.
Min Late 1 = C9
Min Late 2 = C10
Min Late 3 = C11


UNCLASSIFIED / FOUO

 Run #1, with settings of $100 spent on training, 50 jets,
800 employees, and 0% overbooked, was 46.35 minutes
late on the first repetition, 61.92 minutes late on the
second repetition, and 75.18 minutes late on the third
repetition.

$100.00 50 Jets 800 Employees 0% Overbooked

Note: This row of coded variables
were all at their Low (or –1) settings

UNCLASSIFIED / FOUO

 Since we ran our DOE with 3 repetitions, and we want to analyze
the variation in our DOE results, we need to prepare the worksheet
by having Minitab calculate means and standard deviations.
 First, we need to name some blank columns. Name a blank column
StdDev Min Late, a second blank column Count Min Late, and a
third blank column Mean Min Late.


UNCLASSIFIED / FOUO

 Now, we will have Minitab calculate the means and standard
deviations.
 Select Stat>DOE>Factorial>Pre-Process Responses for
Analyze Variability


UNCLASSIFIED / FOUO

 Click on Compute for repeat responses across rows, then click in
the cell under Repeat responses across rows of: and select Min
late 1, Min late 2, and Min late 3 from the columns pane.

 Click in the box Store
standard deviations in and
select the column you named
StdDev Min Late
 Click in the box Store
number of repeats in and
select the column you named
Count Min Late
 Click in the box Store Means
(optional) in and select the
column you named Mean Min
Late
 Click OK

UNCLASSIFIED / FOUO

 Minitab calculates means and standard deviations for each combination of
factors. (Remember: there were 24, or 16, combinations.)
 Minitab also determines the counts. (Remember: there were 3 data points at
each combination, since we ran 3 repetitions at each setting of the DOE.)

Looking at this data Practically, there appears to be some
significance to the factors, but nothing definitive…yet.

UNCLASSIFIED / FOUO

 Before we view the statistics, we always start with the graphs.
 Select Stat>DOE>Factorial>Factorial Plots.


UNCLASSIFIED / FOUO

 Select Main Effects Plot.
 Choose Setup and click
on OK to go to next dialog
box


UNCLASSIFIED / FOUO

 Select StdDev Min Late and Mean Min Late for Responses,
and move all four factors from Available to the Selected box
to have them included in the analysis. Click on OK.

>


UNCLASSIFIED / FOUO

 The Main Effects Plot shows that the number of Employees is the
only driver for Mean Min Late
Main Effects Plot for Mean Min Late
Data Means

Training Dollars Jets
60

45

30

15

0
Mean

-1 1 -1 1
Employees %Overbooked
60

45

30

15

0
-1 1 -1 1

Looking at this data Graphically, it appears that Employees might
be a significant factor influencing the Mean of Time Late

UNCLASSIFIED / FOUO

 The Main Effects Plot shows that the number of Jets AND
Employees are driving the StdDev Min Late
Main Effects Plot for StdDev Min Late
Data Means

Training Dollars Jets
8

6

4

2
Mean

-1 1 -1 1
Employees % Overbooked
8

6

4

2

-1 1 -1 1

Looking at this data Graphically, it appears that Jets AND Employees
might be a significant factor influencing the StdDev of Time Late

UNCLASSIFIED / FOUO

 Now we will run the analysis.
 Select Stat>DOE>Factorial>Analyze Factorial Design


UNCLASSIFIED / FOUO

 Select Mean Min Late and StdDev Min Late as the Response.
 Choose Terms to get to next dialog box


UNCLASSIFIED / FOUO

 Include Terms in the model up through second order (2).
This will include the main effects and two-way interactions.
Click on OK to go back to previous dialog box.


UNCLASSIFIED / FOUO

 Click on Graphs and select the Pareto Chart. Click OK
in both dialog boxes.


UNCLASSIFIED / FOUO

 This chart confirms what we saw earlier in the Main Effects Plot – the
number of Employees has a significant impact on Mean Min Late.
We also see that the interaction term BD* is significant.
This is the „Critical F-statistic‟ used to determine significance.
Pareto Chart of the Standardized Effects
(response is Mean Min Late, Alpha = 0.05)
2.57
F actor N ame
C A Training D ollars
B Jets
BD C E mploy ees
D % O v erbooked
D
AD
BC
Term

B
* - BD is the interaction
A between the factors Jets
AB and %Overbooked. Pareto Chart for
AC Mean Min Late
CD

0 10 20 30 40 50 60 70 80 90
Standardized Effect


UNCLASSIFIED / FOUO

 This chart confirms only part of what we saw earlier in the Main Effects
Plot – the number of Jets has a significant impact on StdDev Min Late
but Employees does not.

Pareto Chart of the Standardized Effects
(response is StdDev Min Late, Alpha = 0.05)
2.571
F actor N ame
B A Training Dollars
B Jets
AD C E mploy ees
D % O v erbooked
C
AC
BC
Term

AB
BD Pareto Chart for
CD
StdDev Min Late
D
A

0 1 2 3 4
Standardized Effect


UNCLASSIFIED / FOUO

 In the Session window, we see that Employees and the interaction
Jets*%Overbooked are the only statistically significant factors for Mean
Min Late. All other main effects and 2-way interactions have a p-value >
0.05.
Factorial Fit: Mean Min Late versus Training Dollars, Jets, ...

Estimated Effects and Coefficients for Mean Min Late (coded units)
This data shows
Term Effect Coef SE Coef T P
Constant 30.42 0.3191 95.34 0.000 Analytically that
Training Dollars
Jets
-0.71
-0.77
-0.35
-0.39
0.3191
0.3191
-1.11
-1.21
0.319
0.279 Employees and
Employees -53.89 -26.94 0.3191 -84.44 0.000 the Jets*%
%Overbooked -1.28 -0.64 0.3191 -2.01 0.101
Training Dollars*Jets 0.68 0.34 0.3191 1.06 0.337 Overbooked
Training Dollars*Employees
Training Dollars*%Overbooked
0.12
1.13
0.06
0.56
0.3191
0.3191
0.18
1.77
0.861
0.138 interaction are
Jets*Employees -0.81 -0.41 0.3191 -1.27 0.259 statistically
Jets*%Overbooked 2.14 1.07 0.3191 3.36 0.020
Employees*%Overbooked -0.10 -0.05 0.3191 -0.16 0.881 significant.

S = 1.27632 PRESS = 83.4049
R-Sq = 99.93% R-Sq(pred) = 99.28% R-Sq(adj) = 99.79%


UNCLASSIFIED / FOUO

 In the Session window, we see that Jets is the only statistically
significant factor for Stdev Min Late. The negative sign for Effect
indicates that standard deviation decreases as Jets increases. All other
main effects and 2-way interactions have a p-value > 0.05.
Factorial Fit: StdDev Min Late versus Training Dollars, Jets, ...
Estimated Effects and Coefficients for StdDev Min Late (coded units)

Constant 4.575 0.7241 6.32 0.001
Training Dollars -0.105 -0.052 0.7241 -0.07 0.945
Jets -5.930 -2.965 0.7241 -4.09 0.009
Employees -2.477 -1.239 0.7241 -1.71 0.148
% Overbooked 0.146 0.073 0.7241 0.10 0.923
Training Dollars*Jets -0.987 -0.493 0.7241 -0.68 0.526
Training Dollars*Employees 2.032 1.016 0.7241 1.40 0.219
Training Dollars*% Overbooked 2.786 1.393 0.7241 1.92 0.112
Jets*Employees 1.655 0.828 0.7241 1.14 0.305
Jets*% Overbooked -0.935 -0.468 0.7241 -0.65 0.547
Employees*% Overbooked 0.932 0.466 0.7241 0.64 0.548

S = 2.89641 PRESS = 429.527

UNCLASSIFIED / FOUO

 Summarizing what we have found in the initial analysis:
 Employees and the interaction between Jets and %Overbooked
had a significant impact on Mean Min Late.
 As seen from the Effects, increasing Jets or %Overbooked
decreases Mean Min Late.
 In addition, when the product of Jets*%Overbooked is positive,
Mean Min Late will increase. If the product is negative, Mean Min
Late will decrease.
Employees -53.89 -26.94 0.3191 -84.44 0.000
Jets -0.77 -0.39 0.3191 -1.21 0.279
%Overbooked -1.28 -0.64 0.3191 -2.01 0.101
Jets*%Overbooked 2.14 1.07 0.3191 3.36 0.020

 Jets had a significant impact on Stdev Min Late. As seen from its
Effect, increasing Jets decreases Stdev.
Jets -5.930 -2.965 0.7241 -4.09 0.009


UNCLASSIFIED / FOUO

 The next step in the DOE analysis is to eliminate
the insignificant terms. This is called “reducing the
model.”
 Every study is different; in this particular case, let‟s
take the following approach:
 Reduce the StdDev model to identify the needed
setting for „Jets‟ since it:
 is the only significant factor influencing StdDev
 plays only a small role in driving the Mean (Coef = -0.39)
 will determine where to set the factor % Overbooked in
the interaction term.


UNCLASSIFIED / FOUO



UNCLASSIFIED / FOUO

1. Click on Terms
2. Remove all Selected Terms: except B:Jets
3. Select OK and OK

2

1 3


UNCLASSIFIED / FOUO

 The mathematical model is taken from the „Coef‟ column of the
Session Window: StdDev Min Late* = 4.575 – (2.965 x Jets)

Factorial Fit: StdDev Min Late versus Jets
Estimated Effects and Coefficients for StdDev Min Late (coded units)

Constant 4.575 0.7792 5.87 0.000
Jets -5.930 -2.965 0.7792 -3.81 0.002

S = 3.11695 PRESS = 177.653

 Conclusion: To reduce StdDev Min Late, we should set the
factor „Jets‟ to the +1 level (55).

UNCLASSIFIED / FOUO

 Now, let‟s reduce the model for the response, Mean.


UNCLASSIFIED / FOUO

1. Click on Terms
2. Remove all Selected Terms: except B:Jets, C:
Employees, D: % Overbooked and the interaction BD.
3. Select OK and OK

2

1 3


UNCLASSIFIED / FOUO

 The mathematical model is taken from the „Coef‟ column of the
Session Window:
Mean Min Late = 30.42 – (0.39 x Jets) – (26.94 x Employees)
–(0.64 x %Overbooked) + (1.07 x Jets x %Overbooked)
Factorial Fit: Mean Min Late versus Jets, Employees, %Overbooked
Estimated Effects and Coefficients for Mean Min Late (coded units)

Constant 30.42 0.3354 90.71 0.000
Jets -0.77 -0.39 0.3354 -1.15 0.273
Employees -53.89 -26.94 0.3354 -80.34 0.000
%Overbooked -1.28 -0.64 0.3354 -1.91 0.082
Jets*%Overbooked 2.14 1.07 0.3354 3.19 0.009

S = 1.34148 PRESS = 41.8811


UNCLASSIFIED / FOUO

 Of the four factors that were investigated, two (Employees and
Jets), plus the Jets*%Overbooked interaction, were significant.
 Jets – to reduce variation, we need to increase Jets to 55.
 Employees - to reduce the average late time from 30 minutes,
we need to increase Employees from 850 to 900.
 Training Budget - had no effect, and can be reduced to $100k
as a budget savings.
 % Overbooked - had marginal effect on time late and on
variation – should be reduced to 0% to improve customer
satisfaction.
 What would you do if this were your organization?


UNCLASSIFIED / FOUO

Is the Change Significant?
1. Conduct a hypothesis test.
2. Open the Capability Data worksheet within the Airline
DOE Data.mpj file.
3. Since we have the baseline sample and the improved
sample, select Stat>Basic Statistics>2-Sample t…


UNCLASSIFIED / FOUO

Is the Change Significant? (continued)
4. Select „Samples in different columns‟
5. Select „Baseline Data‟ for First: and „New Data‟ for Second:
6. Select „Graphs…‟

4
5

6


UNCLASSIFIED / FOUO

7. Select Boxplots of data
8. Click on OK
9. Interpret boxplot. Does there appear to be a graphical
difference?

7 50
Boxplot of Baseline Data, New Data

40

30
Data

20

8
9
10

0

Baseline Data New Data


UNCLASSIFIED / FOUO

10. Review Minitabs Session Window output.
11. Can we state, with 95% confidence, that there is a
statistical difference between our Baseline Data and the
New data? (i.e. Does the Confidence Interval contain „0‟
or is the P-value less than 0.05?)
10

11


UNCLASSIFIED / FOUO

Helicopter Simulation
Phase Two


UNCLASSIFIED / FOUO

Exercise: Helicopter Simulation
 Customers at CHI (Cellulose Helicopters Inc.) have
been complaining about the limited flight time of CHI
helicopters
 Management wants to increase flight time to improve
customer satisfaction
 You are put in charge of this improvement project


UNCLASSIFIED / FOUO

Exercise: Constraints
 Project Mission: Find the combination of factors that
maximize flight time
 Project Constraints:
 Budget for testing = $1.5 M
 Cost to build one prototype = $100,000
 Cost per flight test = $10,000
 Prototype once tested can not be altered
 See allowable flight test factors and parameters on the
next page


UNCLASSIFIED / FOUO

Exercise: Test Factors and Parameters
 Paper Type Regular Card stock
 Paper Clip No Yes
 Taped Body No 3 in of tape
 Taped Wing Joint No Yes
 Body Width 1.42 in 2.00 in
 Body Length 3.00 in 4.75 in
 Wing Length 3.00 in 4.75 in


UNCLASSIFIED / FOUO

Exercise: Roles & Responsibilities
 Lead Engineer – Leads the team and makes final
decision on which prototypes to build and test
 Test Engineer – Leads the team in conducting the test
and has final say on how test are conducted
 Assembly Engineer – Leads the team in building
prototypes and has final say on building issues
 Finance Engineer – Leads the team in tracking
expenses and keeping the team on budget
 Recorder – Leads the team in recording data from the
trials

UNCLASSIFIED / FOUO

Exercise: Phase Two Deliverables
 Prepare a Phase Two Report showing:
 Recommendation for optimal design
 Predicted flight time at optimal setting
 How much money was spent
 Description of experimental strategy used
 Description of analysis techniques used
 Recommendations for future tests
 Comparison of Phase One and Phase Two approaches

Which Team Has The Best Design?


UNCLASSIFIED / FOUO

Takeaways
 Types of experiments – Trial and Error, OFAT, DOE
 Introductory DOE terminology
 Benefits of full factorial vs. fractional designs
 How to use Minitab to design, run, and analyze a DOE
 Use DOE results to drive statistical improvements


UNCLASSIFIED / FOUO

What other comments or questions
do you have?


UNCLASSIFIED / FOUO

References
 Schmidt & Launsby, Understanding Industrial
Designed Experiments


NG BB 47 Basic Design of Experiments

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (20)

Similar to NG BB 47 Basic Design of Experiments

Similar to NG BB 47 Basic Design of Experiments (20)

More from Leanleaders.org

More from Leanleaders.org (20)

Recently uploaded

Recently uploaded (20)

NG BB 47 Basic Design of Experiments