1. Keep It Clean Partnership
Measuring Embraced
Behavior To Evaluate
Effectiveness of MS4
Education and Outreach
StormCon 2012
Session P-35
2. Keep It Clean Partners
Boulder County - 22,469 /44,000
City of Boulder – 97,385
Town of Erie – 18,135
City of Lafayette – 24,453
City of Longmont – 86,270
City of Louisville – 18,376
Town of Superior – 12,483
5. Audiences and Behaviors
Adults
School Children
Construction
Businesses
Municipal employees and contractors
6. Partners for a Clean
Environment
Restaurants
Vehicle Service
Gas Stations
Retail
Facility Managers
Landscapers
Pressure Washers
Municipal
7. Questions about Effectiveness
Questions for the Consultant
Are KICP shared programs effective?
Is KICP program administration efficient?
How can KICP improve water quality and
measure it?
8. Recommendations for Effectiveness
Update IGA and Bylaws
Reinforce watershed approach
Leverage brand awareness
“Keep it Clean”
Define priority pollutants and the behaviors
that generate them
Measure “Embraced Behavior”
9. Embraced Behavior Proposal
“Be the expert about the best practices.” ~
Greg Davis – EPA Region 8
“Benefits can be monetized for MEP.” ~ John
Sorensen, AMEC
“Put the “Point” back into Non-Point Source
pollution education.” ~ Jesse Poore, Felsburg
Holt & Ullevig
10. The Trouble with
“Effectiveness”
“I don’t care if my program is effective. I just
want to be in compliance.” ~ MS4 Operator
If MS4 Education and Outreach Programs are
supposed to accomplish “NPDES” (Pollutant
Discharge Elimination) …
Can we correlate Education to Water Quality?
11. The Trouble with
“Effectiveness”
“We are ‘flying blind’ or, probably, by the
seat of our pants, when it comes to making
decisions about how best to address water
quality problems and allocate our limited
resources for cleanup, pollution prevention
and restoration.” ~ G. Tracy Mehan, former
EPA Assistant Administrator
12. The Trouble with
“Effectiveness”
“Without assessing the effectiveness of the
stormwater management program the
permittee will not know which parts of the
program need to be modified to protect and/or
improve water quality and instead will
essentially be operating blindly.” EPA 2010
Permit Improvement Guide
14. The Trouble with
“Effectiveness”
Peer Research
KPCRC, Washington LCSC, South Carolina
Mindy Fohn Synithia Williams
MRSWMP, California
Heidi Niggemeyer
15. The Trouble with
“Effectiveness”
MRSWMP - Co-permittees, evaluation
structure provided, benchmarks defined
KPCRC - State campaign, local economics,
focus on observing behavior
LCSC - University facilitated education, local
input, targets for audience and pollutants
16. The Trouble with
“Effectiveness”
Literature Research
2001 Study of Six California MS4 Costs
2010 Tracked KICP Expenditures
Category Average KICP
Budget $2.5 Million $1.7 Million
Management $375,000 $362,000
Education $120,000 $48,500
Proportion 4.0% 2.9%
Per Person $0.46 $0.18
Per House $1.24 $0.48
30. What Gets Measured Matters
Typical Permit Language:
“Within [insert deadline, e.g., within the permit
term], the permittee must assess changes in
public awareness and behavior resulting
from the implementation of the program such
as using a statistically valid survey and modify
the education/outreach program accordingly.”
31. What Gets Measured Matters
Permits and guidance typically promote
measuring percent change in behavior.
BUT
Does not correlate behavior to a loading rate
Rarely identifies a valid start point or a
satisfactory end point
Ability to prioritize limited dollars is minimized
32. What Gets Measured Matters
Permit Effectiveness ≠ Behavior Effectiveness
Measuring evaluations and surveys against
learning objectives ≠ Behavior Effectiveness
33. Evaluate Cost Effectiveness
Recent webcasts provide helpful techniques
WEF: Evaluating Stormwater Outreach
EPA: Conducting Effective Stormwater Outreach
Neither promoted direct measurements to
supplement surveys and raised awareness
Neither discussed pollutant loading associated
with behaviors
34. Community Based Social Marketing
Approach by McKenzie-Mohr and Smith
Identify barriers and benefits
Define tools to change behavior
Pilot the approach
Evaluate results
Emphasize direct measurements
Supported by surveys and focus groups
41. Important Question #2
“If you can’t establish a load associated
with a behavior, how are you going to
measure improvement in water quality?”
Hint: Urban runoff loading based on
behavior is uncharted territory
42. Behaviors Impact TMDLs
# of Restaurants Screened: 38
Number Grease Bins Open: 5 (13.2%)
Non-Conforming Bins: 1 (2.6%)
No Bin Seen: 4 (10.5%)
Surface Area Stain
None/Covered: 20 (52.6%)
Slight: 10 (26.3%)
Significant: 4 (10.5%)
Revisit IGA and bylaws to ensure reflection of current goals Revisit priority pollutant focus and associated behaviors to determine strategies - methodically Employ tools to measure embraced behavior - tracking and evaluate resource expenditures ?
When I think about education and outreach, I consider it across all Minimum Control Measures. IDDE, CSW, PCSW, GHPP
I also keep in my mind the 1999 text in the Federal Register regarding Maximum Extent Practicable. EPA envisioned an iterative process that would take 10-15 years to eliminate water quality impairments caused by MS4s In this frame of mind, education and outreach was never expected to be a permanent effort if MCMs were effective Illicit discharges were reported when found and citizens disposed of all waste properly Construction sites were all maintained properly and inspections were self-conducted to identify corrective actions Maintenance facilities were managed in pristine condition with pollution prevention integrated into all operations Watersheds were planned wisely, all developments included water quality treatment and permanent maintenance
The many unique differences between one program and another were fascinating, but frustrating at the same time Each partnership had its own approach and localized resources to approach education and outreach with One unifying theme – credit Mindy Fohn for illustrating this point for me – effectiveness is best defined by direct observations
If we aren’t clear where the end point might be, perhaps we could trend against categorical expenses incurred by other MS4s Trouble comparing due to expense categories and not tracking all things. Did start to get me thinking about budget amounts and the “knee of the curve” argument for CSO LTCPs exponential output of cost to generate an incremental benefit result put that idea into the parking lot for later use
One of these things is not like the other…stormwater is the only mandated effort required to change social and institutional behaviors.
Man’s impact upon the environment is nothing new. We have always been slow to acknowledge our need to embrace best practices in order to get the most out of the land. Hugh Bennett was the first to definitively communicate to the nation that the Dust Bowl was the result of man’s behavior The impact was wide, personal, and deadly – a consequence we may be to far from to appreciate the correlation to stormwater runoff
source: CNN 8/16/2012 http://www.cnn.com/2012/08/16/business/tobacco-health-warnings/index.html?hpt=hp_c2 Even measuring raised awareness isn’t a very good indicator of good behavior. America has one of the highest levels of national awareness about the health effects of smoking. And yet it continues to be the habit of choice for so many. Awareness does not equal embraced behavior.
Guns kill people and people create pollution The Causes of Pollution Conceptually, pollution is NOT the affect of pollutants. Pollution is the result of the ACT of introducing a pollutant into the environment. Therefore, MS4s should focus education on the pollutant generator. With proper behavior, pollution doesn’t occur and permit is not necessary.
Changing behavior is the underlying principle of any public education and outreach effort. Use a variety of assessment methods to evaluate the effectiveness. Are public education and outreach programs changing public awareness and behaviors?
US Stormwater education focuses on surveys and evaluations, but leave direct measurments off hoping that raised awareness will equate changed behaviors McKenzie-Mhor understands that sustainable behaviors are the result of a targeted outreach effort the starts with direct measurements and is supported by indirect assessments.