As a solution, the BER model uses a two-dimensional matrix to aid the evaluation of complex multi-unit programs, with quadrants to identify over and underperforming units. The BER model was inspired by portfolio management approaches from the Boston Consulting Group and the General Electric Grid, as well as quadrant analysis by Andreasen (1995). However, its core principles are based on the concept of social return on investment, where output is always compared to input. It provides a relative perspective on performance that allows evaluators to account for impact based on the resources invested in an initiative.

1. BASIC EFFICIENCY RESOURCE:
A framework for measuring the relative
performance of multi-unit programs
Brian Cugelman, PhD
Managing Director, AlterSpark
4 October 2010
Canadian Evaluation Society's Annual Conference
Optimizing the Practice of Evaluation
Toronto, Canada

2. WHERE BER STARTED
Evaluation of Oxfam GB’s
Global Climate Change
Campaign
Solution to evaluation
challenges
Lot’s of interest
Leitmotiv and AlterSpark
joint publication on BER
2

3. EVALUATION CHALLENGES
Simplifying complex multi-unit programs
Many evaluations are about ROI, but ROI is
difficult to assess in social contexts
Nothing is good or bad, except in comparison to
something else
3

4. THE BER SOLUTION
Provide a simple framework for evaluating complex
multi-component programs, campaigns, or activities
Build on the basic concepts of SROI to evaluate unit's
impact compared to their resources
Offers a relative perspective on performance where
units of analysis are judged in comparison to their
peer units, operating under similar conditions
4

5. THEORETICAL FOUNDATIONS: MATRIX
ANALYSIS FRAMEWORKS
• Boston Consulting Group
• General Electric Grid
• Customer satisfaction quadrant
analysis by Andreasen
• Bloc modeling techniques used by
social network analysts
• Multi dimensional scaling
5

6. THEORETICAL FOUNDATIONS: SROI
SROI: efficiency is output relative to input
• Input constitutes a program’s resources which may
be measured by their budget, number of staff, pool
of talent, social capital, or any measure of capacity,
concrete or abstract
• Output measures a program’s impact, and will vary
according to a program's purpose
6

7. THEORETICAL FOUNDATIONS: SROI
7
Input
High Below Average
Efficiency
Average
Efficiency
Low Average
Efficiency
Above Average
Efficiency
Low High
Output

8. CASE STUDY: OXFAM GB’S GLOBAL
CLIMATE CHANGE CAMPAIGN
8
3.53.02.52.0
Investment
4.5
4.0
3.5
3.0
Impact
Visual stunts & media relations
Effective com at policy events
Using celebrities
Global Oxfam affiliates
Research papers
Rapid news dissemination
Public campaigns/mobilization
Policy analysis
Partnerships with others
Oxfam internal program links
Online campaigning
Media engagement
Lobbying and advocacy
Staff in UNFCCC delegations
Climate hearings
Adopt a negotiator
Perceivedimpact
Perceived resourcing
High
HighLow Low

9. CONDUCTING A BER ANALYSIS
The example in following section is fictional and for illustrative
purposes.
9

10. 1. SELECTING UNITS OF ANALYSIS
10
1. Units of Analysis 3. Perceived Input (Budgets) 2. Perceived Output (impact)
Online engagement •Most
•Average
•Least
•I don't know
•Most
•Average
•Least
•I don't know
Research papers •Most
•Average
•Least
•I don't know
•Most
•Average
•Least
•I don't know
Lobbying and
advocacy
•Most
•Average
•Least
•I don't know
•Most
•Average
•Least
•I don't know

11. 2. MEASUREMENT TOOLS
11
Input Output
• Program budgets (perceived and
real)
• Number of staff
• Number and level of staff
• How often a lobbying keyword
appeared in policy
• Number of widgets produced
• Number of people engaged
• Perceptions of impact
Completely
Disagree
1
2 3 4 5
Completely
Agree
6
I can't say
Online engagement
Research papers
Lobbying and advocacy

12. 3. DATA TYPES
• Quantitative input data may include budgets,
number of staff, or combined multi-dimensional
resource measure
• Quantitative output data may include process
evaluation measures such as the number of
people engaged by a campaign or media hits
• Qualitative measures can include perceived
program investments and perceived output
achieved
12

13. 4. VISUALIZATION APPROACHES
13
High
Low
Low High

14. 4. VISUALIZATION APPROACHES
14
Input
High
• Media relations
• Public relations
• Lobbying and advocacy
• Coalition/partnership
building
Low
• Intra organisational
coordination
• Online engagement
•Research papers
•Public mobilization
Low High
Output

15. 4. VISUALIZATION APPROACHES
15

16. 5. INTERPRETATION
Use BER is as a starting point for deeper discussions
into the performance of units, their challenges,
opportunities, and operating environment
Understand the units of analysis and the informants
who shared their perceptions
Not all units within a program operate under the same
conditions
Some units contribute indirect effects, by
empowering other units
16

17. LIMITATIONS AND RISKS
It is easy to draw conclusions from the simple
visualizations that would never stand in the face
of a deeper understanding of the reality behind
charts
Kotler et al. (2005) noted, reliance on matrix
approaches prompted a number of companies to
sell off strategic assets and plunge into
businesses that they lacked the experience to
manage
17

18. FUTURE WORK AND BER
DEVELOPMENT
Download a copy of the BER whitepaper:
• www.alterspark.com/insights/publications.html
Send comments or examples of BER analyses:
• Dr. Brian Cugelman, AlterSpark
• Eva Otero, Leitmotiv
Share your feedback on Facebook:
• http://www.facebook.com/group.php?gid=143779348990230
18

19. THANK YOU
Brian Cugelman, PhD
Managing Director, AlterSpark
Phone: +1 (416) 921-2055
brian@alterspark.com
www.AlterSpark.com
@AlterSpark alterspark alterspark alterspark
19