Zondits

1. Main Headquarters: 120 Water Street, Suite 350, North Andover, MA 01845 With offices in: NY, ME, TX, CA, OR www.ers-inc.com
MERGE AHEAD:
Integrating Process and Impact Evaluations
2014 ACEEE Summer Study
Patrick Hewlett, P.E.
ERS
August 21, 2014

2. Example: Consider a small-business direct install
program that underwent concurrent but separate
process and impact evaluations.
Process finding:
“The total potential energy savings associated with franchises is
significantly greater than with individually owned businesses. To
better achieve program savings goals, the utilities should engage
franchise accounts with multiple eligible facility locations.”
Impact finding:
“Franchises were found to have higher free-ridership than non-
franchises and should therefore be carefully screened during the
incentive application process.”
WHY INTEGRATE?
8/26/2014 2

3.  Status quo and barriers to integration
 An integrated approach for three phases:
 Evaluation planning
 Data collection
 Reporting
 Joint recommendations:
 Quantified
 Harmonized
 A blueprint for PAs
OVERVIEW
8/26/2014 3

4. STATUS QUO
8/26/2014 4
Traditional Process approach:
• Big-picture assessment of program effectiveness
• Delivery, marketing, outreach, incentive structures,
customer satisfaction
Traditional impact approach:
• Focused on producing one number: net savings
• Detailed file review, site visits with M&V,
engineering analysis
Efficiency program improvement
through recommendations

5.  Timing
 Process evaluations typically occur before impact
 Actionable feedback for PAs, can be implemented more quickly
 Budget
 Two evaluations might not be funded by same source
 Levels of funding are often different
 Additional budget required to allow for strategic planning and joint
recommendation development
 Objective
 Traditional process and impact evaluations not thought to intersect
 However, recommendations benefit from an intersection
BARRIERS TO INTEGRATION
8/26/2014 5

6. Process recommendations
not informed by
comprehensive file
review and field findings
Impact recommendations
often not grounded in
program costs, logistics
CONSEQUENCES
8/26/2014 6
 Result: unrelated, isolated recommendations
 Program hesitant to act on recommendations that
might be out-of-touch or idealistic.

7. AN INTEGRATED APPROACH
8/26/2014 7
Traditional
Process
Approach
Program benchmarking
Interviews with program
staff
Define research objectives
Application review
Interviews with customers,
contractors
Compliance: age, EUL,
cost, payback
Tracking system review
Process findings
Estimation best practices
Implementation
Customer/contractor
experience
Process
recommendations
Integrated
Approach
Benchmarking to define
program needs
Set joint research
objectives
Plan for joint data
collection
Collaborative project
file review
Initial application/
algorithm findings
Joint survey questions
Field data collection on
compliance factors
Discrepancy analysis
driving prioritized, joint
recommendations
based on savings, cost
Traditional
Impact
Approach
Areas of uncertainty
Define research
objectives
Review program
technologies
Project file review
Algorithm review
On-site data collection
and M&V
Interviews with customers,
contractors,
nonparticipants
Savings results
Realization rates
Net-to-gross
Impact discrepancies
Impact recommendations

8. • Program
benchmarking
• Interviews with
program staff
• Define research
objectives
• Benchmarking to
define program
needs
• Set joint research
objectives
• Plan for joint data
collection
EVALUATION PLANNING
8/26/2014 8
Traditional
Process
Approach
Integrated
Approach
Traditional
Impact
Approach
• Areas of
uncertainty
• Define research
objectives
• Review program
technologies

9. • Application review
• Interviews with
customers,
contractors
• Compliance: age,
EUL, cost, payback
• Tracking system
review
• Joint survey
questions
• Collaborative
project file review
• Initial application/
algorithm findings
• Field data
collection on
compliance factors
DATA COLLECTION
8/26/2014 9
Traditional
Process
Approach
Integrated
Approach
Traditional
Impact
Approach
• Project file review
• Algorithm review
• On-site data
collection and M&V
• Interviews with
customers,
contractors,
nonparticipants

10. • Estimation best
practices
• Path to
implementation
• Customer,
contractor
experience
• Process recs
• Discrepancy
analysis driving
prioritized, joint,
actionable recs
based on savings,
cost
REPORTING
8/26/2014 10
Traditional
Process
Approach
Integrated
Approach
Traditional
Impact
Approach
• Savings results
• Realization rates
• Net-to-gross ratios
• Impact
discrepancies
• Impact recs

11.  A key benefit of an integrated approach is quantified
and harmonized recommendations.
 Quantified recommendations
 Allows program to prioritize recommendations by savings
and/or costs
 Impact team can quantify the program’s shortcomings by
savings magnitude, highlighting areas of greatest need
 Process team research informs the costs and barriers of
implementation and proposes solutions
 Result: program administrators able to weigh
recommendations, prioritize by need, and more
immediately act.
AN INTEGRATED APPROACH
8/26/2014 11

12. QUANTIFIED RECOMMENDATIONS
# Projects
Impact on
kWh RR
Impact on
kWh RR
#
Projects
-8%
2%
-8%
0%
-6%
1%
0%
0%
0%
0%
-1%
0%
-1%
0%
-2%
1%
-6%
8%
-2%
3%
-6%
0%
-2%
0%
102 -41% 15% 41
Unsubstantiated savings 5 0
Totals
4 2
Difference in controls 9 0
Estimating
Performance
Incorrect load profile 6 2
Difference in operating hours 17 21
Deemed savings assumed
7 2
Incorrect technology 3 0
2 1
Inspection
Inoperable equipment 1 0
Quantity or size error
Measure
Screening
Ineligible measure 13 0
Incorrect baseline 8 2
Interactivity
Category
Negative Positive
Administrative Administrative error 27 11
8/26/2014 12

13. QUANTIFIED RECOMMENDATIONS
# Projects
Impact on
kWh RR
Impact on
kWh RR
#
Projects
-8%
2%
-8%
0%
-6%
1%
0%
0%
0%
0%
-1%
0%
-1%
0%
-2%
1%
-6%
8%
-2%
3%
-6%
0%
-2%
0%
102 -41% 15% 41
Unsubstantiated savings 5 0
Totals
4 2
Difference in controls 9 0
Estimating
Performance
Incorrect load profile 6 2
Difference in operating hours 17 21
Deemed savings assumed
7 2
Incorrect technology 3 0
2 1
Inspection
Inoperable equipment 1 0
Quantity or size error
Measure
Screening
Ineligible measure 13 0
Incorrect baseline 8 2
Interactivity
Category
Negative Positive
Administrative Administrative error 27 11
8/26/2014 13
Savings
tracking and
measure
screening
issues led to
a 19% drop in
kWh RR

14. QUANTIFIED RECOMMENDATIONS
# Projects
Impact on
kWh RR
Impact on
kWh RR
#
Projects
-8%
2%
-8%
0%
-6%
1%
0%
0%
0%
0%
-1%
0%
-1%
0%
-2%
1%
-6%
8%
-2%
3%
-6%
0%
-2%
0%
102 -41% 15% 41
Unsubstantiated savings 5 0
Totals
4 2
Difference in controls 9 0
Estimating
Performance
Incorrect load profile 6 2
Difference in operating hours 17 21
Deemed savings assumed
7 2
Incorrect technology 3 0
2 1
Inspection
Inoperable equipment 1 0
Quantity or size error
Measure
Screening
Ineligible measure 13 0
Incorrect baseline 8 2
Interactivity
Category
Negative Positive
Administrative Administrative error 27 11
8/26/2014 14
Operating
hours
assumptions
and deemed
savings had
relatively
high swings,
but is action
needed?

15. Process finding:
“Database is fully populated and includes appropriate fields for
tracking applications.”
Impact recommendation:
“Eliminate all duplicate records in the tracking dataset and always
update tracking system with the most current estimates of
savings.”
Integrated recommendation:
“Database design conforms to best practices. Apparently, QC
processes are inadequate because tracking errors degraded the
realization rate by 6%. Convene a team of database administrators,
program implementers, and the contractor to map out the tracking
database and identify specific checkpoints in the system for
reducing these errors.”
HARMONIZED RECOMMENDATIONS
8/26/2014 15

16. HARMONIZED RECOMMENDATIONS
8/26/2014 16
Process finding:
“Customers prove eligibility through extensive screening of project
applications, which drives down participation.”
Impact recommendation:
“Ineligible customers have been approved due to inaccurate
account information as determined through field data collection.
Each project should require an on-site pre-inspection to ensure
eligibility.”
Integrated recommendation:
“Develop a benchmarking system to screen customer-claimed
usage by square footage and facility type. Require field inspection
for those customers that do not meet the benchmark, which is
expected to occur about 15% of the time.”

17.  An integrated evaluation approach requires strategic
planning for joint objectives.
 Assignment of roles, responsibilities
 Agreement on method of valuing recommendations
 Joint interviews with program staff
 Coordination of delivery schedules and check-ins
 Regular communication among teams
 Leading to joint recommendations
 Process research informs feasibility of implementation
 Impact research highlights areas of greatest need
 Result: actionable, grounded recommendations
prioritized by need.
A BLUEPRINT FOR PAS
8/26/2014 17

18. CONTACTS
Patrick Hewlett, P.E.
ERS
phewlett@ers-inc.com
(978) 478-5305
Sue Haselhorst
ERS
shaselhorst@ers-inc.com
(978) 478-5325
8/26/2014 18