CRITICAL THINKING AND
CREATIVE PROBLEM
SOLVING SKILLS

List down things that you expect
to learn from this module

METHODOLOGIES
• Discussion
• Interactive Lecture
• Individual & Group discussions
• Exercise
• Case Study
• Video

At the end of this programme, you will be able to:
 Explain the concept and importance of creative problem solving
and critical thinking
 Explain the key steps in creative problem solving process
 Apply problem solving techniques and tools.
 Use creative thinking techniques to generate creative and
innovative solutions.
OBJECTIVES

 Concept and importance of Problem Solving and Creative Thinking
 Key steps in problem solving process
 How to define and analyse a problem.
 Problem analysis tool
 How to generate creative and innovative solution
 Creative thinking techniques
 How to select the best options
CONTENT OUTLINE

CRITICAL THINKING &
CREATIVE PROBLEM SOLVING
THE PROBLEM SOLVING
METHOD

WHAT IS A PROBLEM?
Kepner Tregoe’s:
A deviation from an expected standard
of performance.
MacCrimmon and Taylor:
A gap between a current and a desired state of
affairs – i.e. a gap between where you are and
where you would like to be.

 Current problem
 Inherited problem
 Stumbled problem
 Hidden problem
 Stated or implied problem (different problems
looked into, different problems found)
TYPES OF PROBLEM ?

 something that shouldn’t have
happened but had happened
 something that should have
happened but did not happen
 something that had happened but
did not fulfill the goals / needs
WHAT IS A PROBLEM ?

 Unknown source
 Within scope of authority
 Needs to investigate
 Based on Quantitative Data
CONCEPT OF PROBLEM IN PDCA
CYCLE
DEMING CYCLE

CREATIVE
- Generative
- Nonjudgemental
- Expansive
When you are thinking
creatively, you are
generating list of new
ideas
CRITICAL THINKING & CREATIVE
PROBLEM SOLVING SKILS
CRITICAL
- Analytical
- Judgemental
- Selective
When you are
thinking critically, you
are making choices
Creative vs Critical

THE IMPORTANCE OF CRITICAL
THINKING & CREATIVE PROBLEM
SOLVING
1. Solving or reducing
personal problems
2. Apply the techniques to assist
friends, spouse and children
to solve their problems
1. Analyze situations in a
systematically way.
2. Resolve problems creatively
3. Assist Superior in solving daily
operation issues & challenges.
4. Developing a pool of young talent
who are problem solvers.
Daily conduct Business Needs

SOFT SKILLS
1. English language
proficiency – oral and
written
2. Knowledge of other
languages an added
advantage
3. Communication skills
4. Thinking skills
5. Planning and
administrative skills
6. ICT skills
COMMUNICATION
SKILLS
1. Able to present ideas
orally and in writing
2. Able to present ideas
in group discussions
3. Able to write reports,
project papers,
meeting minutes
4. Able to write official
letter and
memorandum
5. Able to provide
spontaneous ideas
6. Able to negotiate
7. Able to lead and
manage
THINKING SKILLS
1. Analytical ability
2. Generating critical
ideas
3. Generating creative
ideas
4. Producing high quality
work with minimal
errors
PLANNING &
ADMINISTRATIVE
SKILLS
1. Able to manage time
effectively
2. Able to draw effective
plan of action
3. Able to decide
effectively
4. Able to work in a team
PERSONALITY
1. Self-confident
2. Integrity
3. Trustworthy
4. Resourceful
5. Work smart
ICT SKILLS
1. Information gathering
2. Information
processing
3. Communication
4. Presentation of
information
5. Word processing
WHAT DO EMPLOYERS LOOK FOR?

PERSONAL
Executive
Competency
Model
BUSINESS PEOPLE
OPERATIONS
THE LINK BETWEEN EXECUTIVE COMPETENCY MODEL
AND CRITICAL THINKING & CREATIVE
PROBLEM SOLVING SKILLS
Able to analyze situations & resolve problems creatively

1. THE SIX STEPS APPROACH
Step 1: Presenting Problems
Step 2: Restructure the problem
Step 3: Identify causes to the problem
Step 4: Developing alternative solution
Step 5: Assessing choice of the solution
Step 6: Implementing and evaluating the solution
K.A.I
• Know the problem (1 & 2)
• Analyze the problem (3,4 & 5)
• Implement the solution (6)
THE 6 CREATIVE THINKING
STEPS IN PROBLEM SOLVING

CREATIVE PROBLEM SOLVING PROCESS

PROBLEM SOLVING PROCESS
Construct Overall
Project Plan
Define Process
Define Problem
List Causes
Find Root Causes
Actn. Res. Date
~~~~ ~~~ ~/~/~
~~~~ ~~~ ~/~/~
~~~~ ~~~ ~/~/~
N
Y
Manpower:
Machinery:
Methods:
Materials:
Measurements:
What
When
Where
How much
Brainstorm For
Solutions
Choose Best Solution
& Justify
Trial Implementation
€ •~~~~~~~
•~~~~~~~~
Plan Implementation
Force Field Analysis
Implement
Monitor
Present Plan
Actn. Res. Date
~~~~ ~~~ ~/~/~
~~~~ ~~~ ~/~/~
~~~~ ~~~ ~/~/~

7. Check & Monitor Progress
 Check sheet
 Pareto Diagram
 Radar chart
PLAN
DO
ACTION
CHECK & MONITOR SOLUTION PROGRESS
CHECK
PROBLEM SOLVING PROCESS
1. Project Identification (Problems)
- Brainstorming
- Force ranking
2. Determine causes
- Brainstorming
- Check sheet
- Cause & effect analysis
3. Develop target &
solution
- Brainstorming
- 5W + 1H
4. Data collection
- Checklist
- Stratification
5. Management presentation
- Approval
6. Implement Solution
- Trial
8. If no improvement
- (Relook P2)
9. Slight improvement
- (Relook P3 & P2)
10. Achieve target
- Standardization
(SOP)

Identify and list problems
encountered at or during
working
ACTIVITY

PROBLEM DEFINITION
CRITICAL THINKING & CREATIVE
PROBLEM SOLVING SKILLS

You may need to state the problem in broad terms
since the exact problem may not be obvious.
You may lack information to define it
You can confuse symptoms with underlying causes
Prepare a statement of the problem and find someone
you trust to review it and to talk it over. If the problem is
a job situation, review it with your supervisor or the
appropriate committee or resource.
DEFINING THE PROBLEM

- Clarify the Situation
- Challenge assumptions about the Problem
- Determine possible reasons and evidence
- Explore different perspective concerning the
problem
- Ask more about the original question
DETERMINE WHERE THE PROBLEM
ORIGINATED

Present State
- Write a statement of the situation as it curently
exists
Desired State
- Should include concrete details, contain any
information about possible causes or solutions
DEFINING THE PRESENT STATE
AND THE DESIRED STATE

Q
U
A
L
I
T
Y
TIME
Performance
Gap
Highest
Performance
A
A
B
B
A need or discrepancy between should be condition and
actual situation
DEFINING THE PRESENT STATE
AND THE DESIRED STATE

 Place emphasis on different words in the statement and
ask questions about each emphasis
 Subtitute one word in the statement that explicitly defines
the word to reframe the problem
 Rephrase the statement
 Indicate quantity or time
 Identify and replace or eliminate any persuasive or
opinionated words
STATING AND RESTATING THE
PROBLEM

27
Utilize kipling Tools
5W 1H i.e who, what, where, when, why and
How
ANALYZING THE PROBLEM

ANALYZE SITUATIONS &
RESOLVE PROBLEMS CREATIVELY
CREATIVE : Having the ability to produce new ideas
CRITICAL : Careful attention and judgment
THINKING : A process that acts upon intelligence
INNOVATION : Doing something with the ideas

 What are the possible sources of the problem?
 What are the internal and external changes that
contribute to the particular problem?
 Who is most involved in this problem?
 Do they have the perspective in solving this problem?
 Why does this problem exist?
IDENTIFY CAUSES TO THE PROBLEM
ANALYZING THE PROBLEM

Three sub-steps:
Step A : Identify Potential Causes
Step B : Determine the Most Likely Causes
Step C : Identify the True Root Causes
IDENTIFY CAUSES TO THE PROBLEM
ANALYZING THE PROBLEM

STEP A : Identify Potential Causes
EFFECT
cause
cause
cause
cause
cause
cause
cause
cause
ISHIKAWA DIAGRAM
Main
cause 1
Main
cause 2
Main
cause 3
Main
cause 4
ANALYZING THE PROBLEM

5 accident
cases at the
worksite
No safety features
careless
No proper skills
No safety procedure
cause
cause
cause
Safety not functioning
ISHIKAWA DIAGRAM
Machine Man
Material Method
STEP A : Identify Potential Causes (Example)
ANALYZING THE PROBLEM

PARETO
CHART
STEP B : Determine the most likely causes
ANALYZING THE PROBLEM

WHY-WHY DIAGRA M
Effect of the
Problem
Why?
Why?
Why?
LEVEL 1
LEVEL 2
LEVEL 3
STEP C : Identify the
true root causes
ANALYZING THE PROBLEM

WHY-WHY DIAGRAM
Careless
Sleepy
Sleep
Late
Part
Time
Job
LEVEL 1
LEVEL 2
LEVEL 3
STEP C : Identify the
true root causes
ANALYZING THE PROBLEM

To assess the most frequently-occurring
defects by category
To provide a structured way to collect quality-related
data as a rough means for assessing a process or as
an input to other analyses
To break down (in successive layers of detail)
root causes that potentially contribute to a
particular effect
TOOLS / TECHNIQUES THAT ARE COMMONLY
USED TO SOLVE PROBLEMS
Ishikawa Diagram /
Fish-bone Chart
Check Sheet
Pareto

To compare size of quantity / big or small
numbers.
To show relative proportion
TOOLS / TECHNIQUES THAT ARE COMMONLY
USED TO SOLVE PROBLEMS
To illustrate a project schedule
(star and end date)
Gantt Chart
Pie Graph
Bar Graph

To identify the type of relationship (if any) between two
variables
To roughly assess the probability distribution of a
given variable by depicting the frequencies of
observations occurring in certain ranges of values
To determine whether a process
should undergo a formal examination
for quality-related problems
TOOLS / TECHNIQUES THAT ARE COMMONLY
USED TO SOLVE PROBLEMS
Histogram
Scatter
Diagram
Control Chart

OTHER POPULAR TOOLS / TECHNIQUES COMMONLY
USED TO SOLVE PROBLEMS
Tree Diagram
Affinity Diagram
To breakdown issues, statements and
Ideas until actionable items
Are identify.
Allows large numbers of ideas stemming from brainstorming to
be sorted into groups for review
and analysis
Relationship Diagram
Used for finding solution to problems
that have complex causal relationship

TOOLS / TECHNIQUES TO SOLVE PROBLEMS
ACTIVITY TOOLS
Identify problem Brainstorming, Pareto
Select problem Filtering, Matrix Decision Making
Define problem 5W 2H concept
Analyses problem (find & verify the source) Fish bone diagram, linkage diagram, matrix
verification
Fix a target SMARTER concept
Intervention suggestion Brainstorming, focus group

Identify the root causes of the
problem that you had already
identified.
ACTIVITY

WRITING A PROBLEM STATEMENT
Who - Who does the problem affect?
What - What is the issue? - What is the impact of the
issue?
When - When does the issue occur? - When does it need
to be fixed?
Where - Where is the issue occurring?

• Company profit to increase to at least 21% by
the end of December 2010
• Cases of “Staff Absent Without Leave” among
technical staff to be reduced to at most 10%
per month starting August 2010
• No accident cases at the construction site for
the month of May 2010 onwards
EXAMPLE OF DESIRED STATE
WRITING A PROBLEM STATEMENT

Good Problem
Statement
1. Written
2. Objective
3. Simple
4. No implied causes
5. No implied solutions
WRITING A PROBLEM STATEMENT

• Company profit has dropped from 21% to 17%
from last year
• 10 % of the technical staff were absent without
leave in the month of August 2010
• 5 accident cases were recorded at the plant site
for the month of May 2010
EXAMPLE OF PROBLEM STATEMENT
WRITING A PROBLEM STATEMENT

Example 1 – Problem Statement
 Poorly Written Examples
 “There are too many errors in our reports”
 Good Examples
 “The monthly quality reports contain more than 2 errors on average, greater
than the 0 errors expected”
 Better Examples
 “The past 2 months the quality reports contain more than 2 errors on average,
greater than the 0 errors expected”
 Poorly Written Examples
 “Our delivery time is horrible”
 Good Examples
 “On-Time Delivery has averaged 92%, less than the target of 98.5%”
 Better Examples
 “On-Time Delivery to Customer X been only 92% for the last 4 months, less
than the required 98.5%”

Problem
Statement must
be SMART!
MAJOR
TERM
MINOR TERMS
S Specific Significant, Stretching, Simple
M Measurable Meaningful, Motivational,
Manageable
A Achievable Agreed, Attainable, Assignable,
Appropriate, Actionable, Action-
oriented
R Relevant Realistic, Results/Results-
focused/Results-oriented,
Resourced, Rewarding
T Time-bound Time framed, Timed, Time-based,
Time-boxed, Timely, Time-bound,
Time-Specific, Timetabled, Traceable
E Exciting, Evaluated
R Recorded, Rewarding, Reviewed
WRITING A PROBLEM STATEMENT

a) Develop Problem Statement
b) Identify “Desired State” or Goal
ACTIVITY

GENERATING
SOLUTIONS
CRITICAL THINKING & CREATIVE
PROBLEM SOLVING SKILLS

 Brainstorming - produce as many ideas as
possible, on hold assessment, develop various
solution.
 Basic rules
 On hold assessment
 Facilitate to produce more ideas
 Encourage “free wheeling”
BRAINSTORMING BASICS

WHEN TO USE BRAINSTORMING ?
1. Identify problem in your workplace
2. Analyze a problem
3. Find the probable cause to the problem
4. Find solution to the problem

BRAINWRITING AND MINDMAPPING
Brainwriting is similar to free-association brainstorming
(conducted in silence) through the medium of graphics in
particular, it is classified under the intuitive and progressive
methodologies as it involves driving inspiration from other
members in a cyclical way.
A mind mapping is another method of generating ideas on
paper, but can be conducted alone. A mind map is often
created around a single concept, drawn as an image in the
center of a blank landscape page, to which associated
representations of ideas such as images, words and parts
of words are added. Major ideas are connected directly to
the central concept, and other ideas branch out from those.

THE SIX THINKING HATS
(Edward De Bono)
Facts & information
Feelings & emotions
Being cautious Being positive &
optimistic
New ideas
The Big picture
Blue Hat
Yellow Hat
Red Hat
Green Hat
White Hat
Black Hat

ANALYZNG AND
SELECTING
SOLUTIONS
CRITICAL THINKING & CREATIVE
PROBLEM SOLVING SKILLS

DEVELOPING CRITERIA
Consider the following:
 Ask question: What if…
 Think about what you want the solution to do or not do.
 Think about what values should be considered
The criteria for an effective solution, among others
 Timing – urgent ? If delay?
 Trend – What direction is the problem solving
 Impaact – Is the problem serious?

ANALYSING WANTS AND NEEDS
Wants and Needs seem like a fundamental aspect
of defining the problem.
Needs are items the potential solution absolutely
must meet.
Wants are nice to have items. Provide a weight to
each item to indicate its importance.

USING COST/BENEFIT ANALYSIS
Cost-Benefit Analysis is a method of assigning a
monetery value to the potential benefits of a solution
and weighing those against the costs of
implementing that solution.

STEP 4 : DEVELOPING ALTERNATIVE
SOLUTION
a. Generate a list of possible solutions
b. Determine the best solutions

DEVELOPING ALTERNATIVE
SOLUTIONS
 What are the possible methods for solving the problem?
 What is the impact to every alternative?
 What do you wish to achieve?
 What are the obstacles that prevent us from solving the
problem?
 Among the obstacles, which is the biggest?

PROBLEM TARGET STRATEGIC
SOLUTION
TREE DIAGRAM IN DEVELOPING SOLUTIONS
BRAINSTORMING &
THINK
OUTSIDE THE
BOX
Good impact desired

PROBLEM TARGET
STRATEGIC
SOLUTION
EXAMPLE : TREE DIAGRAM IN SOLUTION FINDING
A Restaurant Owner Finding How To Satisfy Customers
Good impact desired

EXAMPLE : USING TREE DIAGRAM OF PROPOSED SOLUTIONS
TO REDUCE FILTER FAN USAGE BY 30% IN 4Q ‘09
TO REDUCE
FILTER FAN
USAGE BY
30%
IN 4Q ‘09
PURPOSE
(WHY)
METHOD
MATERIAL
MAN
FREQUENT
USAGE
HIGH WEAR
AND TEAR
CHANGE TO
THICKER AND
HARDER
MATERIAL
STANDARD
USAGE NOT
ALLOWED
IMPLEMENT
STANDARD
USAGE
CHEAP
QUALITY
FACTORS
(WHERE)
CAUSES
(WHAT)
COUNTER
MEASURES
(HOW)
RESP
(WHO)
PERIOD
(WHEN)
NO PERSONAL
STORAGE
REUSE
FILTER FAN
CHECK STORAGE
LEVEL
ROKIAH
JASMINE
AUG WK2
TO WK4
FRANCISCA
JASMINE
AUG WK2
AZLINA
AUG WK3
TO WK4
FRANCISCA
ROKIAH
AUG WK3
AUG WK3
WK4
ROKIAH
FARIDAH
Bad impact

Time to
complete
Resources
required
Estimated
cost
How to
implement?
Expected good
impact
Proposed solution
Impact of the
causes
Major Causes
Example : Solution Template

Develop at least 3 solutions from the
root causes identified
ACTIVITY

Factors in selecting :-
 Quantity and quality of information
 Implementation cost (time & RM)
 Easy implementation
 Probability of success if proceed
 Effectiveness of solution
DOING A FINAL ANALYSIS

Three steps:-
 Develop and assign weights to criteria
 Apply the criteria
 Choose the best solution (s)
ASSESSING THE CHOICE OF THE SOLUTIONS
DOING A FINAL ANALYSIS

Effectiveness Efficiency Cost Overall
Rating
Panadol
Take a break
Ginseng
Sleep
Objective: To reduce headache
Scale: Very Negative 1 2 3 4 5 6 Very Positive
5
4
4
5
5
3
3
2
4
2
3
6
14
9
10
13
SELECT THE BEST SOLUTIONS
MATRIX DIAGRAM

Is a method of prioritizing a small number of
workable solutions. The first step is to list all
the possible solutions. Label each solution
with a letter or number.
Next compare the solution in pairs. Decide only
between those two which solution is
preferrable. Assign a number to indicate the
strength of the preferrence for each options.
PAIRED COMPARISON ANALYSIS

Think forward to the solution implementation. Ask
who, what, where, when and How in relation
to implementing the solution.
Brainstorm for potential problems related to the
solution.
ANALYZING POTENTIAL PROBLEMS

 Decision can be based on MUST and WANTS
 MUST - criteria that have to be met without
compromise (must-haves)
 WANTS - criteria desirable to be achieved, would
have tilt the decision (nice-to-haves)
SELECT THE BEST SOLUTIONS
ACTIVITY

PLANNING YOUR NEXT
STEPS
CRITICAL THINKING & CREATIVE
PROBLEM SOLVING SKILLS

STEPS :
 What to do (the activity)?
 When to do (start and finish)?
 Who to do?
 What are the resources required?
PLANNING YOUR NEXT STEP

IDENTIFYING TASKS
PLANNING YOUR NEXT STEP
- Brainstorm with people involved with the problem to
determine the specific steps necessary to make the
solution becomes reality.
- Identify any task that are critical to the timing of the
solution implementation. Critical tasks are items that will
delay the entire implementation schedule if not completed
on time. Non critical tasks are items that can be done as
time and resources permit.

IDENTIFYING RESOURCES
PLANNING YOUR NEXT STEP
Time to think about the resources for making the solution
become reality. The resources are as below:
- Time : How to schedule the project.
- Personnel : Who will complete the identified task.
- Equipment : Any special equipment needed?
- Money : How much will the solution cost
- Information : any additional information required?

IMPLEMENTING, EVALUATING AND
ADAPTING
PLANNING YOUR NEXT STEP
Now is the time to use the Project Management skills to
keep the implementation on track.
During implementation, continue to evaluate the solutions.

ACTIVITY PLAN SEP- 06 OCT-06 NOV-06
W1 W
2
W
3
W
4
W
1
W
2
W
3
W
4
W
1
W
2
W
3
W4
P
Data grouping
1. 5S Awareness Motivation class
2. Use PA system for announcement
3. Announcement made during morning assembly
Management performance unit assembly
D 1. Make all members attend 5S practice motivation class
GROUP A,B,C, and D
C 1. Check with audit
2. Used check sheet every week
A
1. Monitoring – monthly checking from project leaders
2. Understand the marking
3. Audit by the management with standard
IDENTIFYING TASKS
PLANNING YOUR NEXT STEP

ACTIVITY PLANNING
STEP ACTIVITIES
APR-06 MAY-06 JUN-06 JUL-06 AUG-056
W1 W2 W3 W4 W1 W2 W3 W4 W1 W2 W3 W4 W1 W2 W3 W4 W1 W2 W3 W4
Project area
Theme selection
Data Collection
Data Analysis
Solution Evaluation
Implementation
Monitoring result
Standardization
Follow-up Result
Future Plan
P
D
C
A
KEY :
PLAN ACTUAL
PROCEEDING
ADALI S’MAN
YATI
K’RUL FIDA
- INCHARGE - ASSIST
SCH ZUL LAILI MGT SHAM
PROBLEM RESOLUTION
IMPLEMENTATION

IMPLEMENTING THE SOLUTION
a. Divide solution into sequential tasks
ACTION PLAN
Responsible
Person /Group
Action Step
Task/Activity
Begin Date End Date Estimated
Hours
Cost
SOLUTION A
Activity 1
Activity 2
Activity 3
Activity 4
SOLUTION B
Activity 1
Activity 2
Activity 3

a. Divide solution into sequential tasks
ACTION PLAN
Responsible
Person /Group
Action Step
Task/Activity
Begin Date End Date Estimated
Hours
Cost
SOLUTION A
Activity 1
Activity 2
Activity 3
Activity 4
SOLUTION B
Activity 1
Activity 2
Activity 3
IMPLEMENTING THE SOLUTION

A - Divide solution into sequential tasks
To Complete the Job on Time
Responsible
Person /Group
Action Step
Task/Activity
Begin Date End Date Estimated
Hours
Budget
1. Buy a new
PC
Set the
specs
Get the
quotation
Select
vendor
Purchase
2. Train the
Staff
Training
Manual
Organise
Conduct
Training
Abu
Abu
Abu
Abu
12.8.01
21.8.01
16.8.01
22.8.01
15.8.01
20.8.01
22.8.01
25.8.01
3 days
4 days
2 days
4 days
RM0
RM0
RM0
RM5,000
Ahmad
Ahmad
13.9.01
Ahmad
13.9.01
1.10.01
20..9.01
30.9.01
2.10.01 2 days
17 days
7 days RM100
RM2000
RM2000
IMPLEMENTING THE SOLUTION

b. Develop Contigency Plans
Action Step
from Action Plan
Sub-Step Sub-Step Sub-Step
? ?
? ? ? ?
? ?
What will
we do?
What could
go wrong
IMPLEMENTING THE SOLUTION

 Make sure everyone participate
 Build a commitment to action into decision
 Create extensive involvement of those affected
 Build a feedback loop into the process
IMPLEMENTING THE SOLUTION

EVALUATE THE SOLUTIONS
Considerations:-
 Has the desired state achieved?
 Is the process going on smoothly to make sure
no recurrence to problem?

Develop action plan to implement
the chosen solution.
Plan before implementation
ACTIVITY

RECORDING LESSONS
LEARNED
CRITICAL THINKING & CREATIVE
PROBLEM SOLVING SKILLS

PLANNING THE FOLLOW-UP
MEETING
 Must have clear agenda. The Purpose is to conduct a
Final Evaluation of the problem, the selected solution
and the implementation of the project.
 Invite team members involved in the process and the
solution implementation
 Consider the meeting arrangements

CELEBRATING SUCCESS
 Take the time to celebrate the things that went well
during the problem solving process
 Try to recognize each person for their contribution
and accomplishments.
 Also celebrate successes by recognizing the
contributions of Team members in the follow-up
meeting.

IDENTIFYING IMPROVEMENTS
 Take the time to lessons learned and ways to make
improvements so that the next problem solved will be
even better.
 Meeting with Team members and Stakeholders to
identify improvements. It can
- ensures everyone is aware of the challenges
encountered and what was done to resolve
- apply this lessons to future problems and be more
successful

COMMON PITFALLS
 Working on problems that are too general, too large
or not well-defined
 Jumping to solution before really analyse
 Failing to involve critical decision makers
 Tackling problems beyond control
 Failing to develop good reasons of choosing
 Failing to plan how to implement & evaluate