1. Institute for Development Studies
International Workshop
March 26-27, 2013
Impact, Innovation and Learning
Towards a Research and Practice Agenda for the Future
Are development evaluators fighting the last
war?
Robert Picciotto
King's College London
“Evaluation cannot hope for perfect objectivity but
neither does this mean that it should slump into
rampant subjectivity”
Ray Pawson
1

2. The story line
• The past is not always prologue
• The impact evaluation debate is over
• Experimental methods have formidable
advantages… and disadvantages
• Complexity/system thinking: a theory?
• A new development evaluation agenda:
values; human well being; new
development architecture; evaluation
beyond aid
2

3. The risks of “path dependence”
• Past decisions lose their relevance, distort
today’s decisions or impose limits on them:
– What worked yesterday may not work tomorrow
– What did not work yesterday may, with luck, work
tomorrow
• Summative evaluation does not necessarily
translate into formative evaluation: good
evaluators exercise caution when offering
recommendations
• Path dependence also affects evaluation
models and approaches – this is why we are
here!!!
3

4. The impact evaluation debate is over
• Methods have to do with tactics
• Even the best tactics fail without a
strategy
• Attribution analysis of discrete
interventions has dominated recent
debates
• The Stern gang has produced a definitive
“state of the art” review: many
questions beyond attribution need to be
tackled by development evaluators 4

5. The potential of randomisation
• Experimental methods have formidable
advantages: they can help establish causality
by providing a plausible measure of what
results would have been observed had the
intervention not taken place
• This approach is ideal for tackling selection
bias and addresses the counterfactual
dilemma by ensuring that all the other factors
that may affect outcomes are identical except
for stochastic errors
• The statistical significance of findings can be
measured by statistical testing techniques
5

6. Society is not a laboratory
• Redundancy or feasibility issues
• External validity: context matters
• Failure to address the distinctive
accountabilities of partners, replication,
implementation (who? Why? How ?)
• Ethical issues
• Experiments are costly, require superior
skills, large studies, large samples and
specialized quality assurance
6

7. Qualitative approaches contribute to
evaluation rigour
• Without theory there is no credible causal
explanation that adds to development
knowledge
• Understanding of causal relationships calls for
substantive knowledge of the intervention and
its context
• Qualitative methods involve participation,
observation, text based information, village
meetings, open ended interviews, etc.
• Quality can and should be quantified.
• It is also important to figure out why things
did not happen
7

8. Mixed methods together with (or
beyond) the experiment
– Regression and factor analysis
– Quasi-experimental designs
– Multivariate statistical modelling
– Participatory approaches
– Qualitative impact assessment
– Beneficiaries surveys
– General elimination methodology
– Expert panels
– Benchmarking
– Case studies
8

9. Complexity models and systems
thinking: not yet a theory?
• Pluralism and inclusivity in methods
• Networks as privileged units of account
• Interrelationships
• Engagement with multiple actors
• Symbols and diagrams to guide evaluative
inquiry
• From ‘attribution’ to ‘contribution'.
• Rely selectively on all evaluation traditions –
theories of change, realist and critical
evaluation, democratic evaluation, etc.
9

10. Towards a new development
evaluation agenda
A relevant development evaluation
strategy for our troubled times should:
 put values at the very centre of the
strategy
 reflect the new development agenda
likely to emerge beyond the MDGs
 reconsider the ultimate goals of
development cooperation.
10

11. Why should values come first?
• The social arrangements that allow a few
individuals to accumulate enormous wealth
while depriving large segments of the
population of basic necessities have spawned
widespread popular anger and resentment
• The richest 0.5% hold well over a third of the
world's wealth while 68% share only 4%
• Differences in income attributable to effort,
skill or entrepreneurship are not widely
resented – unless the rules of the game are
rigged
• Evaluators should examine these rules! 11

12. Evaluators can no longer sit on the
fence
• Current utilization driven evaluation models are
client-controlled and tend to be value free
• Evaluators should be independent and seek
inspiration from the ethical imperatives of social
justice
• They should master recent policy research
findings about inequality
• They should revise their guiding principles and
metrics in ways consistent with the pursuit of
more equitable, inclusive and environmentally
sustainable growth
• They should give higher priority to the political
dimensions of policies and programs
12

13. We need new metrics
• GNI measures fail to capture services provided
within the household or to reflect environmental
losses, social inequities and human insecurities
• Five planets would be needed to get the world up
to US living standards
• The spectre of global warming is haunting the
poorest and most vulnerable countries
• The reality of poverty encompasses factors that
reach well beyond material deprivation
• The profile of risk management should rise given
the rise of national disasters and the insecurities
of the interconnected global system
13

14. Towards a 3D (McGregor/Sumner)
evaluation model
14
Evaluation
characteristics
Material well-
being
Relational well-
being
Perceptual well-being
Major discipline Economics Sociology Psychology
Dominant evaluation
approach
Cost benefit
analysis
Participatory
evaluation
Empowerment evaluation
Investment focus Physical capital Social capital Human capital
Main unit of account Countries Communities Individuals
Main types of
indicators
Socio-economic Resilience Quality of life

15. Taking account of a new aid
architecture
• Aid is once again an instrument of geopolitics and
new public and private actors have joined the fray
• Aid is increasingly channelled through large,
diffuse and adaptable country based programmes
or through vertical multi-country initiatives
involving multiple actors.
• Aid coalitions are increasingly operating across
borders to tackle “problems without passport”
• The complex political equations that underlie
their operations cannot be solved through
abstract, politically neutral approaches
15

16. Evaluating development cooperation
beyond aid
The footprint of development cooperation is far
more significant than the aid footprint:
• Exports (about $5.8 trillion)- 45 times official
aid flows.
• Remittances ($283 billion) - 2.2 times
• FDI ($594 billion) - 4.6 times as large
• Royalty and licence fees($27 billion) - a fourth
• Huge damage caused by climate change as a
result of OECD countries’ energy policies 16

17. Conclusions
• Even the best methods cannot compensate for
a misguided strategy
• The new development research and
evaluation agenda:
• emphasize ethical standards and democratic
values
• reflect the new human well being paradigm
• come to terms with the new aid architecture
• reach well beyond aid
• experiment with complexity and systems thinking
• promote independence in evaluation processes
• devote more attention to the political dimension 17