1. © 2011 Turoff 11
Foundations of Delphi
for
Foresight and Group Communications
Murray Turoff
turoff@njit.edu
http://is.njit.edu/turoff

2. © 2011 Turoff 2
From: “The People, Yes” by Carl Sandburg
The white man drew a small circle in the sand
and told the red man, “This is what the Indian,
knows,” and drawing a big circle around the
small one, “This is what the white man knows.”
The Indian took the stick and swept an immense
ring around both circles: This is where the
white man and the red man know nothing.”
(A poetic reason for Delphi)
A seer upon perceiving a flood should be
the first to climb a tree – Kahlil Gibran
(Criteria for Study Designers: belief in the results)

3. © 2011 Turoff 33
Definition from 1975
 Design of a group communication process
structured/tailored around the nature of the
application and the nature of the group
 Original paper and pencil rounds
 Anyone can change their view
 Anonymity or pennames
 Scaling theory to promote understanding
 Voting to focus discussion
 Select “knowledgeable” people
 A round took a month – three to five rounds
 Respondents 15 to 500
 Prediction, policy analysis, conditional forecasts,
planning, significance of contributions, new product
characteristics, etc, etc. (book has many examples)

4. © 2011 Turoff 4
Some interesting forecasts
 1953 IBM estimates market for only
50 computers in U.S. with IBM 650
 1969 GE management claims BASIC is
useless
 1975 HP rejects idea of personal
computer before there was one
 1979 Tandy expects to sell only 4,000
of the first real portable computer

5. © 2011 Turoff 5
Single Subjective Estimates
 Personal judgment
 As good as data, models and
information to back it up
 Expert Opinion
 As many examples of wrong ones as
right ones by experts.
 Xerox, Video text, Picture Phone,
X.400
 How do you know which expert to
use?

6. © 2011 Turoff 66
Added definitions and analogs today
 A rose by any other name!
 Recommender systems
 Marketplace Systems
 Collaborative Systems
 Knowledge structures and systems
 Collaborative tagging and Folksonomies
 ESPgame.org A beautifully simple Delphi process
 Structural Modeling (user created models)
 Computer Mediated Communications
 The design of inquiry systems by philosophy
 Scientific: Leibnitz, Locke, Kant, Hegel,
Singer – C. West Churchman book
 Non Scientific! Heidigger (Negotiated reality,
management and marketing)
 Virtual Organizations, teams, groups

7. © 2011 Turoff 7
Online and Real Time Delphi or Asynchronous
Collaborative Systems
 Today this is the future of the Method and its
Evolution
 Many examples already exist and some
experimental evaluation efforts have been
done
 In many of these Anonymity, Pen Names or
Signatures by authors (when they choose) or
by the sponsors is a dynamic option
 As usual in the computer field technological
options are expanding faster than they can
be evaluated except by looking at the results.
 Understanding the successful paper
approachs is still a good guide to how such
systems work

8. © 2011 Turoff 8
Objectives of a Delphi Process
 Promote exploration of a complex problem
 Gathering ideas and information from the
literature and the respondents
 Provide an understanding for the group of the
viewpoints of individuals, subgroups and the
whole group.
 Use of voting and scaling theory
 Expose and Explore disagreements
 Encourage focused discussion
 Encourage participants to focus on what they
know well
 Allow participants to change their minds

9. © 2011 Turoff 9
Paper and pencil vs. On line Dynamic Delphis
 Paper and Pencil
 Specific round structures usually 2 to 5 for paper
and pencil exercises.
 Weeks between rounds
 Much use of paper, scissors, scotch tape, and Xerox
to eliminate duplications
 On Line
 Any individual may deal with the part of the
problem they prefer to work with at the moment
 Computer tracks all communications and knows
what any participant has or has not seen or
responded to
 For Both
 Use of a knowledge structure to capture
information and organize it to avoid information
overload

10. © 2011 Turoff 1010
Delphi does not make decisions
 Delphi should never make a decision, it should be
used to analyze a potential decision or action and
provide that to a single individual or role that is
accountable for the decision to be made.
 The decision maker for real time decision should
always be one of the participants
 In Emergency management many different roles
make decisions that can interfere with one another
and an Emergency System must track the decision
process to insure everyone has access to the most
current relevant information
 This type of system alerts people to conflicts and
those responsible for oversight. The problem solving
Delphi (later) is useful in that type of situation.
 It is unpredictable what Roles (active 24 hours 7 days
a week) are needed at a given time for a given problem
but it is always a group that is needed.
 The person in the role changes dynamically

11. © 2011 Turoff 1111
Characteristics of Delphi I
 A content structure appropriate to the problem being
addressed
 Automatic organization of content contributions
 Dynamic knowledge base creation
 Anonymity of voting and degrees of anonymity for
contributions
 Forced, voluntary, and pen names
 Some groups are told who other members are to
convince them of quality of group (peers)
 Anyone can change their original vote
 Primary goal of votes to expose different views and
generate discussion of causes for that
 Ambiguities, uncertainties, misunderstandings, and
disagreements
 Need to eliminate ambiguities before other issues can
be resolved

12. © 2011 Turoff 1212
Characteristics of Delphi II
 Classifying participants so that votes can be seen
viewed by backgrounds of voters very important for
large studies with very heterogeneous groups
 Use of voting and comments is to encourage
participants to change their views based upon relevant
contributions of others.
 This is a different objective than surveys and
associated assumption about only measuring a
fixed state of mind (psychological characteristics)
that do not change for an adult
 Not all survey guidelines and associated statistics
apply to a process intended to deal with complex
problems and causing people to rethink their
viewpoint
 Pennames extremely useful for allowing continuity of
thought across a set of comments and for execution of
role playing learning games or story telling
 For example: comment on the following policy
resolutions as the company auditor would

13. © 2011 Turoff 13
Motivations for participant
 Want to exchange useful information with other
professionals about a complex problem
 It is a peer group and or people you don’t have
access to
 The facilitators have already gathered all the obvious
material to structure and seed the process
 You are not going to educate them and they will
understand what you have to contribute
 The problem is important, part of the job, and/or does
not have an obvious solution
 It can be a process to test ideas
 It is a consulting task that is paid for
 The sponsor will take the results seriously
 The results will be a useful document for a larger
audience

14. © 2011 Turoff 1414
Ultimate Delphi Goal: Collective Intelligence
 Group problem solving result is better than
any single member could have done before
the discussion process
 Possible with even paper and pencil
 Group can solve problems as fast as any
single member considering the same relevant
materials (asynchronous operation)
 Possible with computers and Structured
Computer Mediated Communications

15. © 2011 Turoff 1515
Delphi Examples:
These have been done very often and lend
themselves to online operation
 Trend Delphi: produces a forecast of a trend along
with the mental model of the group making the
extrapolation of the trend curve into the future
 Problem Solving Delphi: Collects solutions to the
problem which are rescaled to a group interval scale
based upon individuals ranking or paired comparisons.
Use voting to focus discussion on items that need it.
 Policy Delphi: seeks policy resolutions and the
strongest pro and con evidence or arguments to
support each policy resolution
 Cross Impact Modeling: Collaborative building of a
model of the future possible outcomes of a set of
unique events.

16. © 2011 Turoff 1616
Creativity Experiment
Open ended idea generation
 Distribution of total Raw, Unique, and Rare Ideas
 Online Discussion System alone and with Delphi
Voting imposed x group size.
 Problem: product ideas for a pill sized data responder,
ranked list report.
 Notes: Unique ideas (duplications removed), rare ideas
(occurred in 3 or less groups); Statistically significant
on per group and per person that Delphi leads to more
improvisation or creativity.
 Cho, H. (2004) The effect of Delphi structure on small
and medium-sized asynchronous groups, Ph. D.
Dissertation, January 2004, New Jersey Institute of
Technology, Information Systems Department.
http://www.library.njit.edu/etd/index.cfm

17. © 2011 Turoff 1717
Number of Idea Types
Structure x Group Size
44 groups (11 groups in
each condition)
Small
5-7
persons
Medium
11-14
persons
Delphi:
Raw ideas (duplicates) 157 247
Unique ideas 67 111
Rare ideas (g<=3)) 40 57
Unstructured:
Raw ideas (duplicates) 108 192
Unique ideas 52 86
Rare ideas (g<=3) 19 48
Results for total ideas and ideas per person are both statistically
significant. No reduction in performance for individuals in
medium group. (open ended brain storming problem)

18. © 2011 Turoff 1818
Easy Online Delphi Used for Experiment
 Bulletin Board System imposed structure by
monitor
 Good discussion thread structure
 Root items for main ideas
 Replies for discussions of each idea
 Allowed anonymous or real signatures
 Web survey system where monitor collected
and put the ideas for a vote and provided
summaries online
 Once a week for three weeks
 Non structured condition
 Same system but no imposed structure by
monitor and no surveys done

19. © 2011 Turoff 19
Trend Forecasting Delphi Round One
 Provide one or more historical curves of
interest (measures of success or failure,
sales, enrollments, profit)
 Ask for:
 Future projection of a historical curve
 3-5 Assumptions used to draw the curve
 Uncertainties that would cause you to
change the curve you drew
 Assumptions & uncertainties:
 Modifications, competitors actions,
government policies, economic
conditions, investments, etc.

20. © 2011 Turoff 20
Forecasting Delphi Round Two
 Turn ALL into potential assumptions
 One persons assumption is another’s
uncertainty
 Ask for vote on validity:
 Certain, Likely, Maybe, Unlikely,
False
 Show limits of 50 % spread of drawn
projections
 Spread increases with time into the
future

21. © 2011 Turoff 21
1
CERTAIN (Average of 1 to 1.5)
•Low risk of being wrong; Decision based upon this will not be wrong
because of this 'fact’; Most inferences drawn from this will be true.
2
RELIABLE (Average of 1.6 to 2.5)
•Some risk of being wrong; Willingness to make a decision based upon
this; Assuming this to be true but recognizing some chance of error;
Some incorrect inferences can be drawn.
3
NOT DETERMINABLE: (at this time) (Average of 2.6 to 3.5)
•The information or knowledge to evaluate the, validity of this assertion
is not available to anyone --expert or decision maker.
4
RISKY (Average of 3.6 to 4.5)
•Substantial risk of being wrong; Not willing to make a decision based
upon this alone; Many incorrect inferences can be drawn; The converse,
if it exists, is possibly RELIABLE.

22. © 2011 Turoff 22
5
UNRELIABLE (Average of 4.6 to 5)
•Great risk of being wrong; Worthless as a decision basis; The
converse, if it exists, is possibly CERTAIN.
6
NOT PERTINENT/RELEVANT (Used to eliminate some
assumptions from exercise) Even if the assertion is CERTAIN or
UNRELIABLE it has no significance for the basic- issue; It cannot
affect the variable under question an observable amount.
blank
NO JUDGMENT
•No knowledge to judge this item, but the appropriate individual
(expert, decision maker) should he able to provide an evaluation I
would respect.
Validity and/or Confidence Scale

23. © 2011 Turoff 23
Forecasting Delphi Round Three
 Show ordered assumptions from
completely true to completely false
 Ask for modifications to original
projection
 Focus on most important “Maybe”
range
 Ask for significance of impact of assumptions
controllable or those not controllable
 Determine actions organization can take to
force outcome of controllable events
 Determine measures/signals of other external
assumptions becoming true

24. © 2011 Turoff 24
Forecasting Delphi Round Four and “maybe”
Five
 May not be same group, usually more
decision makers in this part of the process
 Determine desirability and feasibility of
actions
 Determine usability of measures or ways to
bring about better intelligence
 Round five for significant disagreements
 Next phase
 Management group to do allocation of
resources to influence or measure things
that can change forecast

25. © 2011 Turoff 25
You have to have a design for all of the process
 Request
 Curve Projection
 Assumptions and Uncertainties
 Turned into Potential Assumptions
 Requested Vote on Validity
 Provide List of Assumptions from True to False
 Request new curve estimate
 Extract Controllable events and Uncontrollable in the
Not Determinable range
 Examine these events for influencing and
measuring
 See Steel and Ferroalloy Delphi article from The
Delphi Method Book for examples

26. © 2011 Turoff 26
Results of Trend Delphi
 A numeric projection and spread which
might not differ from a regression analysis
 A “collaborative knowledge model” of the
groups reasons (assumptions) for the
projection
 A collaborative knowledge model of all the
factors that could change the forecast
 Three rounds
 A plan for:
 Trying to influence the outcome
 Monitoring for surprise changes from
external sources like competitor actions
 Two more rounds

27. © 2011 Turoff 27
Modeling Delphi for the Steel Industry
 Flow diagram by three experts
 45 flow of material legs between different
processes in the industry
 Only 15 legs reported in yearly data
 40 other experts (planners) asked to fill in missing
data for prior year
 Not asked to do anything else
 25 decided to modify the model because they did
not agree with it being the “best” model for them
 Modeling Delphis need more exploration
 Very risky to listen to only one text book, one expert,
or one professor!!!!
 Review the literature always necessary before
design of a Delphi

28. © 2011 Turoff 28
Policy Delphi I
 Major Policy Issue
 List obvious resolutions
 Vote on Desirability and Feasibility
 Allow more proposed
 Show results and disagreements
 Ask for pro, con, and neutral arguments on
resolutions
 Vote on new resolutions
 Vote on arguments for Importance and Validity
 Continue rounds as long as vote changes
are made and new resolutions emerge
 Agreement on underlying comments often lead
to a synthesis or a better resolution
 Not necessary

29. © 2011 Turoff 29
Policy Delphi Scales
Forcing Specificity of Responses
 Policy Resolutions
 Very Desirable, Desirable, Undesirable,
Very Undesirable
 Definitely Feasible, Possibly Feasible,
Possibly Unfeasible, Definitely Unfeasible
 Comments on Resolutions: pro, con, neutral
 Very Important, Important, Slightly
Important, Unimportant
 Certain, Reliable, Risky, Unreliable
 No Judgment always a choice on anything
 Easy to make dynamic

30. © 2011 Turoff 3030
Why Delphi: Human Considerations
 Too many participants needed for face-to-face
 Three to five in any given type of expertise and/or
experiences to ensure all rationales are exposed
 Early experiments by Dalkey on rationales for specific
subjective
 With N types needed for a complex problem this is 3xN to 5xN
participants required
 Different backgrounds that require elimination of ambiguity
and misinterpretations as well as translations of concepts
 Severe disagreements to be mediated
 Freedom of expression and improvisation of ideas without
loss of face
 One out of ten ideas valuable
 High status participants most concerned about this
 Equal Participation allowed
 Minimize group process losses
 An idea can be brought up at anytime

31. © 2011 Turoff 3131
Critical Success Factors
 Composition and quality of the participants
 Results only as good as the people involved
 Compensation for Effort
 Communication with a peer group
 Letting participants nominate others helps (snowball)
 Will learn from others in effort
 Results will serve useful and important purpose
 Consulting pay if above not strong enough
 What is obvious already included, participants used
for what is not obvious (no blank sheets of paper)
 A morphological structure to automatically organize
input material
 Important functions and objectives: exploration,
understanding one another, exposure of issues and
uncertainties, examination of disagreements, and
generation of agreements when possible

32. © 2011 Turoff 3232
Why Delphi: operational challenges
 Subjective judgments required
 Models to support consistency of judgments by
individuals and group
 Building collective models (e.g. cross impact modeling,
structural modeling)
 Producing collective Group views (e.g. Arrow’s paradox and
scaling methods)
 Individual human biases (~10)
 Group process losses and gains (~20)
 Avoiding consensus (Asch & Hawthorne effects) pressure
 Participant effort and shadow time
 Asynchronous flexibility to use any time
 Language or cultural difficulties
 Virtual teams
 Detecting differences by backgrounds for feedback
 Multitasking, Cognitive Limitations, Information Overload

33. © 2011 Turoff 3333
Delphi technology and methods
 Exploring
 Capturing individual knowledge
 Process of design, problem solving, derivation,
knowledge structures
 Animation methods
 Promoting Understanding
 Forming a group synthesis
 Scaling methods
 Feedback to group
 Social judgment & voting theory
 Finalizing Viewpoints (no more changes)
 Evaluation by the group
 Collaborative processes

34. © 2011 Turoff 34
Preventive Security Measures in IS
Management of IS Course (Graduate Students 50% working)
Online system over three weeks
Preventive Measures 72 Total Words (approx.) 25,000
Comments on 433 Average comment size
(words)
480
Pro comments 200 Contributors 20
Con comments 104 Voters 27
Neutral Comments 129 Contributions/person 23
Modifications 25 Contributions/day 25
Total of all items 530 Comments/Measure 6

35. © 2011 Turoff 35
Top and bottom rated preventive measures

36. © 2011 Turoff 3636
Rank Scale Disaster Damage Dimension
20 20.00 Causalities and fatalities
19
18 18.00 Utilities Impact
17
16 16.00
15.90
Potential to spread
Ability of local response adequacy
15 15.43
15.40
15.40
15.38
Loss of command and control
Infrastructure damage
Resources for aid or containment
Time needed to respond
14 14.82 Duration of disaster
13 13.09 Public reaction
12 12.95 Geographic impact
11
10 10.07 Time to return to normal
9
8 8.61 Chance of imminent reoccurrence
7
6
5
4 4.70 Financial Loss
3
2
1
0 0.01 Financial recovery costs
Feedback: a Thurstone Scale for
Relative
Importance of Measures of
Disaster Impact
Graduate Student class Delphi in an
Emergency Management class with
about 30% with real experience in
the area.

37. © 2011 Turoff 3737
Online Delphi’s Today
Dynamic: entering main voting items, voting, making vote changes,
and pro, con, neutral discussion comments, can all go on at once.
 Each individual can focus on what they want to
 System notifies user of changes
 Termination of vote changes is a sign of finalization of the
results
 People not confident about a given item can always not vote or
defer their vote to later
 People can propose rewording (Roberts Rules of Order)
 What took three months can happen in three weeks for complex
problems and large groups
 Straight forward problems (e.g. emergency allocation priorities) can
be done in real time by dispersed virtual groups of 5-20
professionals.
 Typically voting anonymous, comments signed
 Voting really serves to focus discussion on differences of view
 Used as a collaborative learning tool in online class discussions
 System designed as a Social Decision Support System or Dynamic
Delphi

38. © 2011 Turoff 3838
Approaches to the Future
Goals affect what questions to ask
 Automation
 Reduce people time and effort
 Eliminate jobs
 Productivity
 Increase quantity
 Increase quality
 Opportunity (preferred for Foresight
studies)
 Do new things
 Do things differently & Better

39. © 2011 Turoff 3939
Approaches to the Future
Dimensions of Group Communications
 Information Exchange
 Pooling of current data and information
 Cooperation
 Informing of plans and actions/efforts
 Coordination
 Joint planning of actions/efforts
 Collaboration
 Working together on the same
actions/effort
 Too often above words are used as if they
were all the same

40. © 2011 Turoff 40
Possible Delphi Use in Planning
(Continuous Organizational Collaborative Process)
 Problem Solving Delphi
 Pick key variables for analysis
 Trend Analysis Delphi
 Generate Key Potential Assumptions/Events
 Problem Solving Delphi
 Evaluate Important/Significant events
 Cross Impact and ISM
 Develop Integrated Model and Simplify Complexity
 Policy Delphi
 Expose and Investigate disagreements on Decision
and Policy options
 Start at the top again for unresolved situations

41. © 2011 Turoff 41
End of First Talk
Questions or Comments
Carefully
looking at
the future!

42. © 2011 Turoff 4242
Appendix Slides:
Examples of general types of Delphi
and
Dynamic Delphis
 These slides represent more detail on
the specific structures mentioned and
also provide a list of references for
those that want to dig deeper.

43. © 2011 Turoff 4343
Examples of General Delphi Designs
and
Dynamic Delphis
 These have all been done with paper a number of
times
 Some have been done partially or totally online or in
combination
 They could all be done online in a mode where anyone
can work on any part at anytime
 Some of these still require a human monitor to
perform certain functions and some can be fully
automatic
 Users can:
 1. Add new items or content
 2. vote on any item, change their vote, and/or
choose not to vote, and/or vote later after
discussion
 3. Suggest alternative wordings to existing item or
add a new version of an existing item.

44. © 2011 Turoff 4444
System operations
 Allow lists to which options, solutions, goals, etc can
be added
 Allow voting scales for such lists
 Allow comments on list items
 Provide each user with new items of any type and also
any whole discussion they want to see
 Provide vote summaries to indicated the relative
status of voted items and highlight differences
 Determine any different vote patterns by participants
with different expertise and/or experience, and/or
backgrounds
 Show for any item
 Number voting
 Number of vote changes
 Number who might yet vote
 Number of discussion entries about an item

45. © 2011 Turoff 4545
Integrated Collaborative Planning I
1. Problem solving Delphi Process: Determine and
consider the variables that measure the future for any
organizational objective and use the historical data to
allow a projection of both a desirable and undesirable
future that might define the range of uncertainty.
2. Trend Analysis Delphi Process: Use a trend analysis
collaborative process to determine the events that will
influence both the desirable and undesirable
outcomes. Events are any type of policies, actions,
decisions, resource allocations etc that can influence
the future of the key variables.
3. Problem Solving Delphi Process: Evaluate these options
to reach agreements on which are the most important
for influencing the future. Make sure to use a
balanced set that reflects a rational assessment of
what makes the difference between a desirable and an
undesirable future.

46. © 2011 Turoff 46
Integrated Collaborative Planning II
4. Develop a cross impact model for these unique events
and use Interpretive Structural Modeling to take the
subjective judgments of those involved and determine
out of what is potentially a very large event set the
most consistent micro scenarios to reduce this large
event set to a more manageable subset of mini
scenarios.
5. Use these mini scenarios and how they influence one
another to actually build a working model that allows
individuals test the significance of changes in
influencing (general resource allocation) the outcomes
that these scenarios represent. This model can be
treated as a game that professionals can use to test
different alternative decisions and resource
allocations.
6. Use the Policy Delphi to actually resolve
disagreements or judgment uncertainties about
possible decision and policy options. This might
cause restarting the process for some aspect of the
planning problem

47. © 2011 Turoff 4747
Trend Delphi
System Functions Participants Responses Response System Actions
Present a historical trend to be
extrapolated by the
participants
Draw a future curve or
redraw a new one when
a change has occurred
in viewpoint.
Present summary of 50%
median and 0%, 25%,
75% and 100%
boundaries
Request assumptions and
uncertainties used to make
above estimate
Turn all these into potential
assumptions
Vote on validity scale for
each potential
assumption.
Scale is from completely
true to completely false.
Reorder assumptions from
true to false.
Focus on middle range
(maybe) and ask which
can be influenced or
measured for occurrence
Assume these can reduce the
future uncertainty in the
curve
Ask for a redrawing of curve
extrapolation based upon
assumption list for each
trend curve in the study
Supply suggestions on how
to influence or measure
the maybe assumptions
causing significant
uncertainty in the
projected curve.
Summarize important
findings at any time:
Trend, true and false
assumptions, assumptions
that cause uncertainty,
and their potential
actions, and
measurements

48. © 2011 Turoff 4848
Problem Solving Delphi
System Functions Participants Responses Response System Actions
State the problem and request
solution options
Provide options to solve
the problem
Present options in order of
occurrence
Request paired comparisons to
measure individual
preferences for options
Make comparisons for
option pairs that a
participant feels
confident about
judging at any time.
Use Thurstone's law of
comparative judgment
(using incomplete
information) to derive a
single group interval
scale.
Calculate uncertainty due to
those who have not yet
voted with same type of
scale.
Show interval scale.
This indicates disagreements
when two or more items are
close together. This also
shows clustering. Ask for
comments about items
where people disagree with
current position of an item.
Make comments about
items you want to see
others change their
votes about.
Present discussions about
items for review. As
more people vote or
change votes scales will
reflect decreasing
uncertainty and often
more separation between
options.

49. © 2011 Turoff 4949
Policy Delphi
System Functions Participants
Responses
Response System
Actions
State a policy issue to be
examined. Ask for specific
policy solutions
Add resolution options or
specific policies
Request vote for Desirability
and Feasibility scales of
each solution
Plot two dimensional
distribution of policy
resolutions
Exploring desirable but
infeasible solutions often
important
Request comments
especially about those
showing disagreement
Request comments about
policy resolutions.
Indicate if comment is
pro, con, or neutral.
Request vote on comments for
importance and validity
It might be considered
important because others
believe it to be true
A person may think a
comment is important
because others think it
is valid.
Do same two dimensional
plots and summarize
discussions

50. © 2011 Turoff 5050
Cross Impact Modeling I
System Functions Participants Responses Response System Actions
Use problem solving Delphi to
produce a set of future unique
events focused on a given
situation
Evaluate those events for
their relative
importance to the
future objective
guiding the choice of
events
Place the final most important
events into a cross impact
model
Ask each individual to answer:
What are the probabilities of each
event occurring in some future
time frame?
Tell them for each event that they
should assume it will or will
not occur and ask them to
express any changes in the
probabilities of the other
events due to that certain
knowledge about the future.
Show them the expected
outcome of their
model which will have
differences from their
predictions. Allow
them to vary initial
probabilities to see
how the future
changes.
Allow them to go back
and modify some of
their estimates
Create the cross impact model
using the approach by Turoff
(logistic, Fermi Dirac
equations). This provides a
scale changes from nonlinear
probabilities (0 to 1) to a
linear influence factor
between each pair of events
(plus to minus infinity).
When participants are satisfied
with their individual model
utilize the internal linear
influencing factors (Cij) to
create a group model.

51. © 2011 Turoff 5151
Cross Impact Modeling II
Creating Scenarios
System Functions Participant Responses Response System
Actions
Analyze the internal parameters
to show people which of the
relationships between which
events show the most
disagreement among the
group.
Ask for comments on these
combinations form those
who have inconsistent or
extreme views.
Show these comments and
others and allow those
who wish to change
some of their original
estimations.
Create a model of
interacting scenarios by
voting on where to stop
the integration of the
events in process that
can turn all the events
into one scenario
When no more changes are
being made produce.
Use Interpretive
Structural modeling to
generate a set of macro
scenarios collecting
individual events that
are tightly coupled into
a set of scenarios that
interact.
Requires human monitor to
know when to trigger
the scenario creation
part

52. © 2011 Turoff 5252
1972 Paper Example
The Delphi Method Book
Event
number
Description
1
The U.S. gets in a trade war with one or more of its major trading partners (Japan, Canada, western European
countries).
2
Comprehensive Tax Revision S Enacted With Most Present exemptions And Exclusions Removed, But With
Rates Lowered.
3 Rigorous anti-pollution standards are adopted and strictly enforced for both air and water.
4 The U.S. averages at least 4 percent per year growth rate of real GNP for the time frame.
5 Defence spending declines steadily as a percent of the federal government's administrative budget.
6
The U.S. Experiences At Least One Major Recession (GNP Decline is greater than 5 Percent for a duration greater
than 2 quarters) during the ten year period.
7
A federal income maintenance system (e.g., negative income tax) replaces essentially all current state and local
welfare programs.
8 The oil import quota system is phased out and domestic oil prices allowed to fall to the world price.
9 The U.S. agricultural price support system is dismantled.
10
A federal-state and local revenue-sharing program is adopted which allocates at least 5 percent of federal revenues
to state and local governments.
Table 1 Events

53. © 2011 Turoff 5353
Use of Interpretative Structural Modeling
with Cross Impact Analysis
Each cycle is a mini scenario or possible new single event

54. © 2011 Turoff 54© 2009 Turoff 54
Cross Impact Additions
 Decide which events you can replace
with micro scenarios to reduce event
complexity.
 Add objective and goals as event sinks
 Add source events as investments
 Develop offense and defense events to
create a game between two players or
two teams
 Can model natural and man made
disasters for exploration and training

55. © 2011 Turoff 5555
The following slides contain references
for those that want to learn more

56. © 2011 Turoff 5656
Papers on my website 1
 Complete copy of original reference book on the Delphi Method
 The Delphi Method: Techniques and Applications by Harold
Linstone and Murray Turoff, 1975. (http://is.njit.edu/turoff)
 Turoff, M., Hiltz, S. R., Li, Z., Wang, Y., Cho, H., Yao, X.,
(2004),Online Collaborative Learning Enhancement through the
Delphi Method, Proceedings of the OZCHI 2004 Conference,
November 22-24, University of Wollongong, Australia
 Murray Turoff, Starr Roxanne Hiltz, Michael Bieber, Ajaz Rana
(1998), Collaborative Discourse Structures in Computer Mediated
Group Communications, 1998
 Turoff, Murray, and Starr Roxanne Hiltz (1996), Computer Based
Delphi Processes, a version will appear as an INVITED BOOK
CHAPTER for Michael Adler and Erio Ziglio, editors., Gazing
Into the Oracle: The Delphi Method and Its Application to Social
Policy and Public Health, London, Kingsley Publishers (in press).

57. © 2011 Turoff 57
Papers on my website 2
 Turoff, Murray (1997) Alternative Futures for Distance Learning:
The Force and the Darkside, The material in this paper was
utilized for an Invited Keynote Presentation at the UNESCO /
OPEN UNIVERSITY International Colloquium, April 27-29:
Virtual Learning Environments and the Role of the Teacher, Open
University, Milton Keynes. It also forms the basis of a planned talk
at the Third International ALN (Asynchronous Learning
Networks) meeting in NY City, October, 1997. (Presentation
Overheads for ALN conference, October, 1997, NYU)
 Turoff, Murray (1995), Software Design and the Future of the
Virtual Classroom, Journal of Information Technology for
Teacher Education, Vol 4, No. 2, 1995
 Turoff, Murray (1999), An End to Student Segregation: No More
Separation Between Distance Learning and Regular Courses. A
summary of the invited plenary for the Telelearning 99 meeting in
Montreal, Canada, November, 1999. (Also: ppt presentation used
in talk.)

58. © 2011 Turoff 5858
Other Papers 1
 Turoff, M., Hiltz, S.R.: The Future of Professional Communities of
Practice. In: Weinhardt, C., Luckner, S., Stößer, J. (eds.) WeB
2008. LNBIP, vol. 22, pp. 144-158. Springer-Verlag, Berlin
Heidelberg (2009)
 Xiang, Y, Turoff, M., and Chumer, M. Designing a group support
System to Review and Practice Emergency Plans in Virtual Teams,
Proceedings of the 6th International ISCRAM Conference,
Gothenburg, Sweden, May 2009 (http://iscram.org)
 White, Connie, Murray Turoff, Bartel Van de Walle, A Dynamic
Delphi Process Utilizing a Modified Thurstone Scaling Method:
Collaborative Judgment in Emergency Response, Proceedings of
ISCRAM 2007, 4th International Conference on Information
Systems for Crisis Response and Management, Delft, the
Netherlands, May 13-16, Brussels University Press
 Plotnick, Linda, Elizabeth Avey Gomez, Connie White, Furthering
Development of a Unified Emergency Scale Using Thurstone's Law
of comparative Judgment: A progress Report, Proceedings of
ISCRAM 2007, 4th International Conference on Information
Systems for Crisis Response and Management, Delft, the
Netherlands, May 13-16, Brussels University Press

59. © 2011 Turoff 59
Other papers 2
 Turoff, Murray, Starr Roxanne Hiltz, Xiang Yao, Zheng Li,
Yuanqiong Wang, and Hee-Kyung Cho, Online Collaborative
Learning Enhancement Through the Delphi Method, Turkish
Online Journal of Distance Education-TOJDE April 2006 ISSN
1302-6488 Volume: 7 Number: 2 Article: 6, Publisher: Anadolu
University, Eskisehir, Turkey,
http://tojde.anadolu.edu.tr/index.htm
 Hee-Kyung Cho, Murray Turoff, and Starr Roxanne Hiltz, The
Impact of Delphi Communication on Small and Medium Sized
Asynchronous Groups: Preliminary Results, HICSS 36, January
2003, IEEE Computer Society Press.
 Turoff, M., Hiltz, S. R., Li, Z., Wang, Y., Cho, H. "The Delphi
Process as a Collaborative Learning Method." In (edited by J. C.
Moore) Elements of Quality Online Education: Into the
Mainstream: Wisdom from the Sloan Consortium, 121-134.
Needham, MA: Sloan-C, September 2004
 Banuls, V., and Turoff, M., Scenario Construction via Cross-
Impact, Draft under review 2009.

60. © 2011 Turoff 6060
Other Papers 3
 Cho, H.K. & Turoff, M., “Delphi Structure and Group
Size in Asynchronous Computer-Mediated
Communications,” Proceedings of the Americas
Conference on Information Systems, Tampa, August
2003.
 Wang, Y., Li, Z., Turoff, M. and Hiltz, S.R. (2003).
Using a social decision support system toolkit to
evaluate achieved course objectives. Proceedings of the
Americas Conference on Information Systems, Tampa,
August. (Nominated as a “best paper.”)
 Turoff, Murray and S. R. Hiltz, (1995), Computer
Based Delphi Processes, in Michael Adler and Erio
Ziglio, editors., Gazing Into the Oracle: The Delphi
Method and Its Application to Social Policy and Public
Health, London, Kingsley Publishers, pp. 56-88.

61. © 2011 Turoff 61
Other papers 4
 Worrell, W., Hiltz, S. R., Turoff, M. and Fjermestad, J. (1995) An
experiment in collaborative learning using a game and a computer-
mediated conference in accounting games. Proceedings of the 28th
Annual Hawaii International Conference on System Sciences, Vol.
IV, pp. 63-71. Los Alamitos, CA: IEEE Computer Society Press,
1995.
 Hsu, Enrico Y. P., Hiltz, S. R., and Turoff, Murray (1992).
Computer-Mediated Conferencing System as Applied to a Business
Curriculum: A Research Update. In V. S. Jacob and H. Pirkul,
eds., The Impact of Information Technology on Business Schools:
Research, Teaching and Administration, Proceedings of the 20th
Annual North American Conference of the International Business
School Computer Users Group, pp. 214- 227. Awarded "Best
Paper- Teaching.“
 Hiltz, S.R. and Turoff, M., The Network Nation: Human
Communication via Computer, 1978, revised edition reprinted
1993 by MIT Press