1. Linking Evaluation and Cost-Benefit Analysis
in Criminal Justice: A Practical Introduction
March 24, 2011
Elizabeth Drake, Senior Research Associate, Washington State Institute for Public Policy
Valerie Levshin, Policy Analyst, Cost-Benefit Analysis Unit, Vera Institute of Justice
Slide 1

2. Welcome
Linking Evaluation and Cost-Benefit Analysis
in Criminal Justice: A Practical Introduction
The webinar will start at 2pm.
Call 1-866-740-1260 for the audio portion of the
webinar.
Slide 2

3. Linking Evaluation and Cost-Benefit Analysis
Criminal Justice: A Practical Introduction
Elizabeth Drake Valerie Levshin
Slide 3

4. The Cost-Benefit Knowledge Bank for Criminal Justice (CBKB) is a
project of the Vera Institute of Justice funded by the U.S.
Department of Justice’s Bureau of Justice Assistance.
• Website (cbkb.org)
• Cost-Benefit Analysis Toolkit
• Snapshots of CBA Literature
• Podcasts, Videocasts, and Webinars
• Roundtable Discussions
• Community of Practice
Slide 4

5. What You Will Learn Today
• Why evaluation is an important element of a
cost-benefit analysis (CBA).
• Different ways to assess program/policy impacts.
• What meta-analysis is and how it can be used in
a CBA.
Slide 5

6. Today’s Agenda
Introduction and Housekeeping 5 minutes
The Role of Evaluation in CBA 5 minutes
Ways to Assess Program/Policy Impacts 10 minutes
Meta-Analysis and CBA 30 minutes
Q&A 10 minutes
Slide 6

7. Housekeeping items
Questions
 Use the chat feature to send us your
questions at any time during the webinar.
 We will address your questions after
each section of the presentation.
Slide 7

8. Housekeeping items
Webinar support and troubleshooting
 Call: (800) 843-9166
 Email: help@readytalk.com
This webinar is being recorded
The recording and PowerPoint will be posted to
cbkb.org
Slide 8

9. The Role of Evaluation in
Cost-Benefit Analysis
Slide 9

10. What is Cost-Benefit Analysis (CBA)?
• A tool to assess the pros and
cons of policies and programs
• A method for finding out what
will achieve the greatest net
benefit to society
• An approach to policymaking
Slide 10

11. CBA in Five Steps
1. Determine the impact of the initiative
2. Determine whose perspectives matter
3. Measure costs
4. Measure benefits (in dollars)
5. Compare costs and benefits
Slide 11

12. Evaluation and CBA
• CBA is only possible if there is
information about program/policy
impact.
• If you don’t know the outcome,
you can’t measure the benefits
and some of the costs.
• You need to evaluate the initiative or draw on research
to predict program/policy outcomes.
Slide 12

13. Ways to Assess Program and
Policy Impacts
Slide 13

14. Ways to Assess Program/Policy Impacts
• Evaluate the initiative
 Comparison group design
 Random assignment design
• Draw on evaluations of similar initiatives
 Literature-based estimate
 Meta-analysis
Slide 14

15. Comparison Group Design
Compare the outcomes for people in your program to a
similar group of people not in your program
 For example: matching groups, pre-post.
 Need to match groups to make sure they’re similar.
 Need to consider outside factors (such as the economy) that
could influence the outcome.
 If the groups are similar, and other factors are ruled out, then
we can assume that the program affected the outcome.
Slide 15

16. Comparison Group Example
Example: CBA of Washington’s
Drug Offender Sentencing
Alternative (DOSA)
 WSIPP’s evaluation compared the
recidivism rates before/after DOSA was implemented in 1999.
 Evaluation results: DOSA reduced recidivism for drug offenders,
not drug-involved property offenders.
 CBA results: benefits > costs for drug offenders.
Benefit / Cost ratio = $7.25 - $9.94 for drug offenders
Benefit / Cost ratio = $0.93 for drug-involved property offenders.
Slide 16

17. Random Assignment Design
Conduct a randomized assignment study to assess the
impact
 Participants are randomly assigned into program or “control”
groups.
 Best way to create very similar groups, where the only difference
between them is program participation.
 Differences in outcomes can be attributed directly to the
program.
 e.g., if program participants have lower recidivism rates that
non-participants, then we know the program reduced recidivism
rates.
Slide 17

18. Random Assignment Example
Example: CBA of the Center for
Employment Opportunity (CEO)
 MDRC random assignment evaluation
showed that CEO reduced recidivism
rates.
 CBA showed that recidivism reduction
generated taxpayer, victim and offender benefits.
Benefit / Cost ratio = about 3 to 1
Slide 18

19. Make a Literature-Based Estimate
• Review the literature to determine the impact of similar
initiatives
• Example: CBA of Raising the Age in North Carolina
 How will trying 16- and 17-yr-olds in the juvenile instead of the adult
system affect their recidivism rates?
 6 studies show that the recidivism rates are 0%-50% lower than in
the adult system.
 CBA assumes that trying youth in the juvenile system will reduce
recidivism rates by 10%.
 Sensitivity analysis shows how using a different recidivism reduction
affects CBA results.
Slide 19

20. Meta-Analysis
• Review the literature to estimate the average effect of
a program/policy on outcomes
• Review all evidence
• Give more weight to results of stronger evaluations
• Details are up next
Slide 20

21. Which Approach to Use?
• Depends on time, resources, staff expertise
• Aim for stronger research designs: the more rigorous the
evaluation, the more accurate the CBA results
• Random assignment, comparison group evaluation are
sometimes unfeasible; need to draw on the literature instead
Increasing Random assignment
difficulty, Comparison group
time, and Meta-analysis
expertise
Literature-based estimate
Slide 21

22. Questions
Slide 22

23. Meta-Analysis and CBA
Slide 23

24. Washington State Institute for Public Policy
(WSIPP)
Washington legislature has asked WSIPP this question:
Are There Evidence-Based Policy Options That Improve Public Outcomes,
but at Less Cost?
Exhibit 4
Reducing Crime With Evidence-Based Options: What Works, and Benefits & Costs
WSIPP
Washington State Institute for Public Policy Effect on Crime Benefits and Costs
Estimates as of October, 2006 Outcomes (Per Participant, Net Present Value, 2006 Dollars)
.
Percent change in crime Benefits to Benefits to Costs Benefits (total)
Notes: outcomes, & the number of Crime Victims Taxpayers (marginal program Minus
“Consumer Reports” Lists:
"n/e" means not estimated at this time. evidence-based studies on (of the reduction (of the reduction cost, compared to Costs
Prevention program costs are partial program costs, pro-rated to which the estimate is based in crime) in crime) the cost of (per participant)
match crime outcomes. (in parentheses) alternative)
(1) (2) (3) (4) (5)
Programs for People in the Adult Offender System
Vocational education in prison -9.0% (4) $8,114 $6,806 $1,182 $13,738
Intensive supervision: treatment-oriented programs -16.7% (11) $9,318 $9,369 $7,124 $11,563
General education in prison (basic education or post-secondary) -7.0% (17) $6,325 $5,306 $962 $10,669
Cognitive-behavioral therapy in prison or community -6.3% (25) $5,658 $4,746 $105 $10,299
Drug treatment in community -9.3% (6) $5,133 $5,495 $574 $10,054
Correctional industries in prison -5.9% (4) $5,360 $4,496 $417 $9,439
Drug treatment in prison (therapeutic communities or outpatient) -5.7% (20) $5,133 $4,306 $1,604 $7,835
Adult drug courts -8.0% (57) $4,395 $4,705 $4,333 $4,767
Employment and job training in the community -4.3% (16) $2,373 $2,386 $400 $4,359
Electronic monitoring to offset jail time 0% (9) $0 $0 -$870 $870
Sex offender treatment in prison with aftercare -7.0% (6) $6,442 $2,885 $12,585 -$3,258
Intensive supervision: surveillance-oriented programs 0% (23) $0 $0 $3,747 -$3,747
Washington's Dangerously Mentally Ill Offender program -20.0% (1) $18,020 $15,116 n/e n/e
What Works?
Drug treatment in jail -4.5% (9) $2,481 $2,656 n/e n/e
Adult boot camps 0% (22) $0 $0 n/e n/e
Domestic violence education/cognitive-behavioral treatment 0% (9) $0 $0 n/e n/e
Jail diversion for mentally ill offenders 0% (11) $0 $0 n/e n/e
Life Skills education programs for adults 0% (4) $0 $0 n/e n/e
What Doesn’t?
Programs for Youth in the Juvenile Offender System
Multidimensional Treatment Foster Care (v. regular group care) -22.0% (3) $51,828 $32,915 $6,945 $77,798
Adolescent Diversion Project (for lower risk offenders) -19.9% (6) $24,328 $18,208 $1,913 $40,623
Family Integrated Transitions -13.0% (1) $30,708 $19,502 $9,665 $40,545
Functional Family Therapy on probation -15.9% (7) $19,529 $14,617 $2,325 $31,821
Multisystemic Therapy -10.5% (10) $12,855 $9,622 $4,264 $18,213
Aggression Replacement Training -7.3% (4) $8,897 $6,659 $897 $14,660
Teen courts
Juvenile boot camp to offset institution time
Sex offender cognitive-behavioral treatment
Restorative justice for low-risk offenders
Interagency coordination programs
-11.1%
0%
-10.2%
-8.7%
-2.5%
(5)
(14)
(5)
(21)
(15)
$5,907
$0
$32,515
$4,628
$3,084
$4,238
$0
$8,377
$3,320
$2,308
$936
-$8,077
$33,064
$880
$205
$9,208
$8,077
$7,829
$7,067
$5,186
What Can Give Washington
Juvenile drug courts -3.5% (15) $4,232 $3,167 $2,777 $4,622
Regular surveillance-oriented parole (v. no parole supervision)
Juvenile intensive probation supervision programs
Juvenile wilderness challenge
Juvenile intensive parole supervision
Scared Straight
0%
0%
0%
0%
+6.8%
(2)
(3)
(9)
(10)
(10)
$0
$0
$0
$0
-$8,355
$0
$0
$0
$0
-$6,253
$1,201
$1,598
$3,085
$6,460
$58
-$1,201
-$1,598
-$3,085
-$6,460
-$14,667
Taxpayers a Good Return
Counseling/psychotherapy for juvenile offenders -18.9% (6) $23,126 $17,309 n/e n/e
Juvenile education programs
Other family-based therapy programs
Team Child
Juvenile behavior modification
Life skills education programs for juvenile offenders
-17.5%
-12.2%
-10.9%
-8.2%
-2.7%
(3)
(12)
(2)
(4)
(3)
$41,181
$15,006
$5,759
$19,271
$6,441
$26,153
$11,231
$4,131
$12,238
$4,091
n/e
n/e
n/e
n/e
n/e
n/e
n/e
n/e
n/e
n/e
(Better Outcomes) for Their Money?
Diversion progs. with services (v. regular juvenile court) -2.7% (20) $1,441 $1,034 n/e n/e
Juvenile cognitive-behavioral treatment -2.5% (8) $3,123 $2,337 n/e n/e
Court supervision vs. simple release without services 0% (8) $0 $0 n/e n/e
Diversion programs with services (v. simple release) 0% (7) $0 $0 n/e n/e
Juvenile intensive probation (as alternative to incarceration) 0% (5) $0 $0 n/e n/e
Guided Group Interaction 0% (4) $0 $0 n/e n/e
Prevention Programs (crime reduction effects only)
Nurse Family Partnership-Mothers -56.2% (1) $11,531 $8,161 $5,409 $14,283
Nurse Family Partnership-Children -16.4% (1) $8,632 $4,922 $733 $12,822
Pre-K education for low income 3 & 4 year olds -14.2% (8) $8,145 $4,644 $593 $12,196
Seattle Social Development Project -18.6% (1) $1,605 $4,341 n/e n/e
High school graduation -10.4% (1) $1,738 $2,851 n/e n/e
Guiding Good Choices -9.1% (1) $570 $2,092 n/e n/e
Parent-Child Interaction Therapy -3.7% (1) $268 $784 n/e n/e
Program types in need of additional research & development before we can conclude they do or do not reduce crime outcomes:
Programs needing more research for people in the adult offender system
Case management in the community for drug offenders
COSA (Faith-based supervision of sex offenders)
Day fines (compared to standard probation)
Domestic violence courts
0% (13)
-22.3% (1)
0% (1)
0% (2)
Comment
Findings are mixed for this broad grouping of programs.
Too few evaluations to date.
Too few evaluations to date.
Too few evaluations to date.
Given the Current Level of
Credible Research, What Don’t
Faith-based programs 0% (5) Too few evaluations to date.
Intensive supervision of sex offenders in the community 0% (4) Findings are mixed for this broad grouping of programs.
Medical treatment of sex offenders -21.4% (1) Too few evaluations to date.
Mixed treatment of sex offenders in the community 0% (2) Too few evaluations to date.
Regular parole supervision vs. no parole supervision 0% (1) Too few evaluations to date.
Restorative justice programs for lower risk adult offenders 0% (6) Findings are mixed for this broad grouping of programs.
We Know?
Therapeutic community programs for mentally ill offenders -20.8% (2) Too few evaluations to date.
Work release programs (from prison) -4.3% (4) Too few recent evaluations.
Programs needing more research for youth in the juvenile offender system
Dialectical Behavior Therapy 0% (1) Too few evaluations to date.
Increased drug testing (on parole) vs. minimal drug testing 0% (1) Too few evaluations to date.
Juvenile curfews 0% (1) Too few evaluations to date.
Juvenile day reporting 0% (2) Too few evaluations to date.
Juvenile jobs programs 0% (3) Too few recent evaluations.
Juvenile therapeutic communities 0% (1) Too few evaluations to date.
Mentoring in juvenile justice 0% (1) Too few evaluations to date.
Slide 24

25. Meta-Analysis and CBA: Overview
What I’ll cover today
• Meta-analytical concepts
 Research design quality
and discount factors
• Cost-benefit concepts
 Resources we monetize
What I won’t cover today
• Cost-benefit data and
calculations
• Meta-analytic calculations
Slide 25

26. Meta-Analysis
Slide 26

27. What is Meta-Analysis?
Empirical Summarization of a Set of Literature
• Meta-analysis produces an average effect on something.
• The unit of measurement is an effect size, which measures the
degree to which a program has been shown to change an
outcome for program participants relative to a comparison
group.
• Not all research is of equal quality, and this greatly influences
the confidence that can be placed in the results of a study.
• A meta-analysis is only as good as the selection and coding
criteria used to conduct the study.
Slide 27

28. Meta-Analytic Procedures:
Important Criteria to Determine Inclusion of Studies
1. Search Criteria (published and unpublished sources)
2. Comparison group studies
• no single group, pre/post research designs
3. Intent-to-treat sampling procedures
• Completers only = bias treatment effect
4. Crime outcomes
• Prefer dichotomous outcomes
• Longest follow-up period
• Felony convictions
Slide 28

29. Meta-Analytic Procedures:
Standards of Rigor and Adjustments to Effect Sizes
An adjustment factor is assigned to the results of individual
effect sizes based on our judgment concerning the research
design quality.
Rating Research Design Quality Discount
Multiplier
5 Random assignment None
4 Random assignment with issues .75
3 Studies that attempt to statistically control for .75
unobserved factors (e.g., regression
discontinuity or natural experiment)
2 Well done comparison group study with many .625
controls
1 Less well implemented with some covariates .5
Slide 29

30. Meta-Analytic Procedures:
Standards of Rigor and Adjustments to Effect Sizes
Discount for:
• Not “Real World” evaluations
• Lab settings
• Evaluation researcher is the program developer
• Weak outcome measure (i.e., incarceration)
Internally consistent set of procedures.
Slide 30

31. Questions
Slide 31

32. Meta-Analysis and
Cost-Benefit
Slide 32

33. Cost-Benefit Procedures
Once we have an effect size, how much does it cost to buy
that effect size, and what’s it worth to achieve it?
CJS resource
Effect size response and Cost-benefit
Relative to base victimizations
population results
incurred
Slide 33

34. Results: What Works to Reduce Crime?
Change In Benefits Minus Costs,
Crime per-person, life cycle
Adult Offenders (# of EB Studies) (Probability: you lose $)
Cog-Behavioral Treatment -4% (27) $7,100 (<1%)
Correctional Education -5% (13) $7,700 (<1%)
ISP: surveillance -2% (23) -$2,900 (≈53%)
ISP: treatment -18% (11) $6,200 (≈13%)
Juvenile Offenders
Functional Family Thpy (wf) -14% (8) $23,000 (<1%)
Aggression Repl. Trng (wf) -9% (4) $12,900 (<1%)
(Draft 2010 Results)
Slide 34

35. Pulling It All Together
Results, results, and more results.
1. Meta-analysis and cost-benefit analysis can inform
stakeholders where resources are best utilized
2. But results will only tell you the average effect
3. Follow up with an outcome evaluation to ensure you
are getting the results you expect
An example in Washington.
Slide 35

36. Questions
Slide 36

37. Wrap-Up
Slide 37

38. Recap of Today’s Webinar
You learned:
• Why evaluation is an important element of a
cost-benefit analysis (CBA).
• Different ways to assess program/policy
impacts.
• How meta-analysis can be used in a CBA.
Slide 38

39. Follow-up
Please complete the evaluation form as you leave this
training.
To receive information and notifications about upcoming
webinars and other events
• Visit the Cost-Benefit Knowledge Bank for Criminal Justice at
http://cbkb.org.
• Subscribe to receive updates from CBKB.
• Follow us on Twitter at http://www.twitter.com/CBKBank.
The next webinar will focus on discussing cost-benefit
results with the media. Stay tuned for updates.
Slide 39

40. This project is supported by Grant No. 2009-MU-BX K029 awarded by the Bureau of
Justice Assistance. The Bureau of Justice Assistance is a component of the Office of
Justice Programs, which also includes the Bureau of Justice Statistics, the National
Institute of Justice, the Office of Juvenile Justice and Delinquency Prevention, and
the Office of Sex Offender Sentencing, Monitoring, Apprehending, Registering, and
Tracking. Points of view or opinions in this document are those of the author and do
not represent the official position or policies of the United States Department of
Justice.
Slide 40

41. Contact Information
Elizabeth Drake
ekdrake@wsipp.wa.gov
(360) 586-2767
Valerie Levshin
vlevshin@vera.org
(212) 376-3062
cbkb@cbkb.org
http://www.cbkb.org
Slide 41

42. Thank you!
Slide 42