This document discusses experimental and quasi-experimental designs. It outlines the key components of classical experimental designs, including independent and dependent variables, experimental and control groups, pretesting and posttesting. It also discusses threats to internal and external validity and variations like quasi-experimental designs that use nonequivalent groups or time series when randomization is not possible. Quasi-experiments aim to make groups as comparable as possible through matching or using natural cohorts.
2. OUTLINE
Introduction
The Classical Experiment
Experiments and Causal Inference
Variations in the Classical
Experimental Design
Quasi-Experimental Designs
3. 3
•Experimentation is an approach to research
best suited for explanation and evaluation
•An experiment is “a process of observation,
to be carried out in a situation expressly
brought about for that purpose”
•Experiments involve:
•Taking action
•Observing the consequences of that action
•Especially suited for hypothesis testing
4. 4
•Variables, time order, measures, and groups
are the central features of the classical
experiment
•Involves three major pairs of components:
•Independent and dependent variables
•Pretesting and posttesting
•Experimental and control groups
5. 5
• The Independent Variable takes the form of a
dichotomous stimulus that is either present or
absent
• It varies (i.e., is independent) in our
experimental process
• “The Cause”
6. 6
• The outcome, the effect we expect to see
• Depends on the Independent Variable
• Might be physical conditions, social behavior,
attitudes, feelings, or beliefs
• “The Effect”
7. 7
• Subjects are initially measured in terms of
the Dependent Variable prior to association
with the Independent Variable (pretested)
• Then, they are exposed to the Independent
Variable
• Then, they are re-measured in terms of the
Dependent Variable (posttested)
• Differences noted between the
measurements on the Dependent Variable
are attributed to influence of the
Independent Variable
8. 8
• Experimental group – Exposed to whatever
treatment, policy, initiative we are testing
• Control group – Very similar to experimental
group, except that they are NOT exposed
• If we see a difference, we want to make sure
it is due to the Independent Variable, and
not to a difference between the two groups
9. 9
• Pointed to the necessity of control groups
• Independent Variable: improved working
conditions (better lighting)
• Dependent Variable: improvement in
employee satisfaction and productivity
• Workers were responding more to the
attention than to the improved working
conditions
10. 10
• We often don’t want people to know if they
are receiving treatment or not
• We expose our control group to a “dummy”
Independent Variable just so we are treating
everyone the same
• Medical research: Participants don’t know
what they are taking
• Ensures that changes in Dependent Variable
actually result from Independent Variable
and are not psychologically based
11. 11
• Experimenters may be more likely to
“observe” improvements among those who
received drug
• In a Double-Blind experiment, neither the
subjects nor the experimenters know which is
the experimental group and which is the
control group
• Broward County Florida and Portland, Oregon
domestic violence policing units study:
“keeping safe” strategies
12. 12
• First, must decide on target population –
the group to which the results of your
experiment will apply
• Second, must decide how to select
particular members from that group for
your experiment
• Cardinal rule – ensure that Experimental
and Control groups are as similar as
possible
• Randomization purposes towards this
13. 13
• “Randomization”
• Central feature of the classical experiment
• Produces experimental and control groups
that are statistically equivalent
• Farrington and associates:
• “Randomization insures that the average unit
in the treatment group is approx. equivalent
to the average unit in another group before
the treatment is applied”
• “All Other Things are Equal”
14. 14
• Experiments potentially control for many
threats to the validity of causal inference
• Experimental design ensures:
• Cause precedes effect via taking posttest
• Empirical correlation exists via comparing
pretest to posttest
• No spurious 3rd variable influencing
correlation via posttest comparison between
experimental and control groups, and via
randomization
15. 15
• Conclusions drawn from experimental
results may not reflect what went on in
experiment
3. History: External events may occur during
the course of the experiment
4. Maturation: People constantly are growing
5. Testing: The process of testing and
retesting
16. 16
4. Instrumentation: Changes in the
measurement process
5. Statistical regression: Extreme scores
regress to the mean
6. Selection biases: The way in which subjects
are chosen (use random assignment)
7. Experimental mortality: Subjects may drop
out prior to completion of experiment
17. 17
8. Causal time order: Ambiguity about order of
stimulus and Dependent Variable – which
caused which?
9. Diffusion/Imitation of treatments:
Experimental group may pass on elements to
Control group when communicating
10. Compensatory treatment: Cgroup is
deprived of something considered to be of
value
18. 18
11. Compensatory Rivalry: Control group
deprived of the stimulus may try to
compensate by working harder
12. Demoralization: Feelings of deprivation
among control group result in subjects
giving up
19. 19
• Potential threats to internal validity are only
some of the complications faced by
experimenters; they also have the problem
of generalizing from experimental findings
to the real world
• Two dimensions of generalizability:
• Construct Validity
• External Validity
20. 20
• Concerned with generalizing from experiment
to actual causal processes in the real world
• Link construct and measures to theory
• Clearly indicate what constructs are
represented by what measures
• Decide how much treatment is required to
produce change in Dependent Variable
21. 21
• Significant for experiments conducted under
carefully controlled conditions rather than
more natural conditions
• Reduces internal validity threats
• John Eck (2002): "diabolical dilemma."
• Suggestion:
• explanatory studies internal validity
• applied studies external validity
22. 22
• Becomes an issue when findings are based
on small samples
• More cases allows you to reliably detect small
differences; less cases result in detection of
only large differences
• Finding cause-and-effect relationships
through experiments depends on two related
factors:
• Number of Subjects
• Magnitude of posttest differences between the
experimental and control groups
23. 23
• Four basic building blocks present in
experimental designs:
2.The number of experimental & control groups
3.The number & variation of experimental stimuli
4.The number of pretest & posttest
measurements
5.The procedures used to select subjects and
assign them to groups
• Variations on the classical experiment can be
produced by manipulating the building blocks
24. 24
• When randomization is not possible for legal
or ethical reasons
• Renders them subject to Internal Validity
threats
• Quasi = “to a certain degree”
• Two categories:
• nonequivalent-groups designs
• time series designs
25. 25
• When we cannot randomize, we cannot
assume equivalency; hence the name
• We take steps to make groups as comparable
as possible
• Match subjects in Experimental and Control
groups using important variables likely
related to Dependent Variable under study
• Aggregate matching – comparable average
characteristics
26. 26
• Cohort – Group of subjects who enter or leave
an institution at the same time
• Ex: A class of police officers who graduate
from a training academy at the same time,
All persons who were sentenced to
probation in May
• Necessary to ensure that two cohorts being
examined against one another are actually
comparable
27. 27
• Longitudinal Studies
• Examine a series of observations over time
• Interrupted – Observations compared before
and after some intervention
(used in cause-and-effect studies)
• Instrumentation threat to internal validity is
likely because changes in measurements
may occur over a long period of time
• Often use measures produced by CJ
organizations
28. 28
• A large number of variables are studied for a
small number of cases or subjects
• Case-oriented research: Many cases are
examined to understand a small number of
variables (Boston Gun Project)
• Variable-oriented research: A large number of
variables are studied for a small number of
cases or subjects
• Case Study Design: Centered on an in-depth
examination of one or a few cases on many
dimensions
