There are no silver bullets or magic solutions to WASH-related problems. What works in one place, does not necessarily fit the context in another. And for solutions to be effective, often broader systemic change is needed that goes beyond the solution itself. This poses difficult questions for assessing the potential for innovations to WASH service delivery. Randomized Control Trial (RCT)-like approaches to test new approaches are often simply not relevant or feasible. Alternative forms of generating evidence are needed. Moreover, appropriate use of evidence in learning and decision-making processes is needed in order to reach scale. This presentation was shared during the IRC lunch meeting 'The Use of Evidence', 6 March 2013.

1. The use of evidence in achieving
IRC‟s mission

2. Background
• IRC‟s vision and mission – action oriented – a think/do tank – a
change agent
• How does “evidence” fit into that? What is “evidence”?
• How does evidence fit into our vision of a learning and adaptive
WASH sector?
• What role do we have vis-à-vis collecting and using evidence?

3. A global movement around “evidence based
approaches”
• Driven by a critique of endless cycles of implementation that don‟t
seem to go anywhere
• Of „development by anecdote‟; development by story-telling
• Move away from ideology-based interventions
• Rise of business informed „metrics‟
• Rise of “big data” – which can be mined to identify patterns of
evidence
• Rise of „randomistas‟
• Rise of „evidence‟

4. Evidence?
• But what is evidence? And what role does it play in improved WASH services?
– Measurement of impact – WB DIME – supporting iterative learning.
Strengthening capacity to do impact evaluation
– Randomistas – Randomised Controlled Trials – the „gold standard‟ of
(medical) research – physician envy?
• At worst – a reductionist approach that focusses on „the evidence‟ rather than
„the intervention‟ – looking at what works and what doesn‟t and not at the
underlying causes and processes
• An approach that risks missing complex and non-linear relationships in
development „space‟
• But …. a real opportunity for a knowledge broker … or a think tank. The
question is not „evidence: yes or no‟ – but “what evidence – for what process”

5. A „development lab‟/RCT approach
Fine in practice – but:
• Where did the „idea‟ come from?
• Who owns it?
• Who owns/actions the results?
• What happens when it doesn‟t work?
Intervention Control
• The development lab is an group group
international development lab
Any significant
difference?

6. Our approach
• Often we do not know beforehand, which “innovation” to test – these need to emerge from the context
• Experiments are rooted into a country context
• Ownership of ideas is local (at least partially)
• Scaling happens because people „buy‟ the idea
• But… in the testing there may be space for RCTs, or similar methods, as well as a range of other forms of
generating and using evidence
• Our development lab is a national development lab!

7. Accept the possibility of failure!

8. Evidence at multiple different scales and as
part of multiple different processes
Evidence of problems: Evidence of Evidence from
to helps kick-start a successes elsewhere: experiments/pilots: to
process to help stimulate feed into process
ideas
But.. difficult to have
evidence of a concept
beforehand

9. But also ….
Evidence that the whole process is functioning …….
Evidence that the whole process is leading to results: better longer lasting
services.
This is critical to IRC – it is proof that what we do …. works!

10. But what sort of evidence?
• Despite what randomistas say ….
– There is a huge range of evidence …. A broad evidence
spectrum
– From purely qualitative to hard quantitative
– From case studies to surveys/samples and …RCTs
– There is NO one gold standard. Evidence is context specific.
– Evidence is expensive
– Generating evidence must be subject to the same cost-benefit
analysis as any other intervention

11. Where is IRC‟s strength?
• Long history of qualitative/soft work – including case-studies
• Strong in mixed methods: quantifying the qualitative (QIS, MPA), but
also trials with text analysis, e.g. SenseMaker and Infolution
• Recently, rapid rise in more quantitative work linked to LCCA and service
delivery indicators
• Gaining experience at measuring outcomes of process work in a more
rigorous way (process documentation, SenseMaker – failed; QDA)
• But limited/no real experience of measuring impact (although new tools
present us with the means of doing so)

12. Programme/
process
focused
QDA
Country/district
IRC reporting key metrics
Case studies
Process
documentation
QIS etc. LCCA,
SDI etc.
Project/pilot
focused
Qualitative Mixed Quantitative

13. Use of evidence: communicating evidence
for change
• How to present evidence so that:
– It is problem/solution focused (not just descriptive)
– Provides clear suggestions for policy/practice
– Is comprehensible, useful and actionable by target audiences
• How to tell our story about evidence so that:
– We do not simply come across as defensive towards
randomistas (and fellow travelers)
– Demonstrate a real appreciation of the use of evidence in real
change processes

14. Discussion
• What type of (new) evidence is most important for IRC‟s work
• What type of communication/medium is best suited to communicating
this?
• Where does IRC most need new capacity to develop and use
evidence?
• Group into and discuss for:
– International (DP, INGO)
– National (Government: technocrats but also politicians and ministry
of finance)
– District/Local

15. Where does IRC need to develop new
strengths?
• More trials with qualitative methods a la QDA and text analysis
• Quantitative methods for generating/interpreting evidence at
national, district, and project level
– Unlikely to do our own RCTs – but need to understand them and
their roll in wider evidence discussion
• Logic of EF is that district level is crucial
• But, communications also critical