1. www.wateraid.org
Webinar on final evaluation
and impact assessment of
Governance and Transparency
Fund Programme
Tuesday 23rd
April 2013

2. www.wateraid.org
Costa Rica
Honduras
Nicaragua
Guatemala
Burkina Faso
Ghana
Mali
Nigeria
Ethiopia
Kenya
Madagascar
Malawi
Uganda
Zambia
India
Bangladesh
WA/FAN GTF programme

3. www.wateraid.org
Background and purpose of the two
exercises
• The evaluation is
primarily for
Accountability and the
impact assessment is
primarily for Learning
• Complementary
exercises
• CPs and partners are
primary users of results
• Results for
communication,
fundraising etc
• Progress against the
baseline data is critical

4. www.wateraid.org
Overview of all evaluation and learning
processes and how they link together
 Mid Term Review – WHAT so far?
 Evaluation – WHAT?
 Impact assessment – So WHAT?
 Learning review – HOW?
 Most significant change
analysis – the WHAT about the SO
WHAT?

5. www.wateraid.org
Key Stakeholders in the process
DFID/KPMG

6. www.wateraid.org
Which country programme is doing what

7. www.wateraid.org
Different levels and how to deal with
this
• 7 countries are doing a full scale evaluation
• 9 countries are doing small scale evaluation
• All countries are doing an impact
assessment except from Kenya
• Small scale = updating Mid Term Review
• Full Scale = in depth assessment based on
key areas

8. www.wateraid.org
Length of the consultancy and how to
use your time
• Total number of days = 25 to be shared between two
exercises
• Rough Guideline:
Step 1: Understanding the context - understanding of the
problem in country that GTF is addressing
• Background reading - 1 day
• Working with country prog staff and key informants - reinforcing
understanding of programme, stakeholders and intervention design,
findings of MTR (if there was one) and conclusions - up to 3 days
Step 2: Enquiry – conducting self-assessment, semi-
structured interviews, FGD…..8 days
Step 3: Analysis - Self-assessment collation of results,
coding of qualitative data….2.5 accountability and 2.5 days
for learning analysis;
Step 4: Write the first draft of the report - 4 days
Step 5: Revisions and redraft of the report - 4 days

9. www.wateraid.org
Timeline
Dates Actions
4th
April ToR for Evaluation and Impact Assessment sent out to all countries
19th
April Each country to sign contracts with local consultants
23th April Webinar with all consultants
May 17th
Local consultants to submit draft Impact assessment section of
report
May 31st
Local consultants to submit draft Evaluation reports
June 24th
Local consultants to submit final Evaluation report including Impact
assessment
July 11th
MoF to submit global consolidated impact assessment
Week of July 22nd
CM to submit draft Evaluation report and share report to KPMG
Week of July 29th
Last Annual learning meeting
Mid Sept CM to submit Final Global Evaluation report
End of Sept /mid
October
Papa to share report with KPMG
End of October Submission of WaterAid PCR to KPMG

10. www.wateraid.org
The difference between an
evaluation and impact
assessment

11. www.wateraid.org
In summary....
Questions Monitoring Evaluation Impact Assessment
Why do we do
it?
Measures on-going
activities
Measures performance
against objectives
Assesses change in peoples
lives
What is the
main focus?
Focus on programme
interventions
Focus on programme
interventions
Focus on stakeholders
At what level? Outputs Outcomes/impact Impact and change
What are the
key questions
to ask?
•What is being done?
•Is our programme
progressing as
planned?
•What happened? Did
we achieve what we set
out to achieve in terms
of:
•Effectiveness
•Efficiency
•Relevance
•Sustainability
•Impact
•So what actually
changed?
•For whom?
•How significant is it for
them?
•Will it last?
•What, if anything, did our
programme contribute?

12. www.wateraid.org
The Evaluation Component
Catherine Currie

13. www.wateraid.org
GTF Summative Evaluation
• We are conducting a critical analysis of
the GTF Programme in order to assess
whether or not it achieved its goals
 Whether the planned activities occurred
 Whether the activities led to achievement of goals;
 How effective the project was;
 How costly the project was; etc.
• This is a summative or end of programme
evaluation

14. www.wateraid.org
Purpose of evaluation
• For accountability- to enable
beneficiaries, board members, etc to know
how funds have been used;
• Our country evaluations will assess:
– objectives against logframe targets and
milestones;
– programme performance by OECD DAC
criteria of effectiveness, efficiency, relevance,
sustainability, replicability and impact

15. www.wateraid.org
A useful Global Evaluation
Report• Top tips include:
– i) the process of collation, analysis and write up
– ii) enhanced rigour and comparability of results and
reports…………...so
• A consistent stance
• Support and advice through online forum
• Verification of evidence
• Step-by-step validation of evaluation results
• Quality assurance processes
• Prioritise the use of country systems
• Use a set of agreed working definitions for key terms
• Use the WaterAid report template

16. www.wateraid.org
Evaluation Questions
Relevance:
• What we expect: Details of the programme’s significance
with respect to increasing voice, accountability and
responsiveness within the local context.
Evaluation Questions:
1. How well did the programme relate to governance priorities at
local, national or internal levels? Please demonstrate with
examples in relation to: i)increasing voice; ii) accountability;
and, iii) responsiveness within the local context.
2. How well did the programme relate to the Country Strategy
Paper aims and objectives? Of WaterAid and where
applicable of the FAN network – ie regional secretariats and of
DFID.
3. How logical is the current theory of change?

17. www.wateraid.org
Effectiveness
• What we expect: An assessment of how far the
intended outcomes were achieved in relation to
targets set in the original logical framework.
Evaluation Questions:
1. Have interventions achieved the objectives? At
country regional and global level.
2. How effective and appropriate was the programme
approach? How effective was the MEL system and
framework?
3. With hindsight, how could it have been improved?

18. www.wateraid.org
Partnership
• What we expect: How well did the
partnership and management
arrangements work and how did they
develop over time? Please consider areas
such as monitoring, evaluation and
learning arrangements. If possible,
consider from a regional perspective.

19. www.wateraid.org
Advocacy
• What we expect: To what extent has GTF
contributed to WaterAid influencing
targets?
Evaluation Questions:
1.How has the programme helped
implement successful advocacy
strategies? Are there any lessons learned
about measuring influencing.
2.How has the programme contributed to
the overall in country advocacy strategy?

20. www.wateraid.org
Equity
• What we expect: Discussion of social differentiation (e.g. by
gender, ethnicity, socio economic group, disability, etc) and
the extent to which the programme had a positive impact
(from an accountability perspective) on the more
disadvantaged groups.
Evaluation Questions:
1. How did the programme actively promote gender equality?
2. What was the impact of the programme on children, youth
and the elderly?
3. What was the impact of the programme on ethnic minorities?
4. If the programme involved work with children, how were child
protection issues addressed?
5. How were the needs of excluded groups, including people
with disabilities and people living with HIV/AIDS addressed
within the programme?

21. www.wateraid.org
Value for Money
• What we expect: Good value for money is the optimal use of
resources to achieve the intended outcome.
Evaluation Questions:
1. Has economy been achieved in the implementation of
programme activities?
2. Could the same inputs have been purchased for less money?
3. Were salaries and other expenditures appropriate to the
context?
4. What are the costs and benefits of this programme?
5. Is there an optimum balance between Economy, Efficiency
and Effectiveness? Overall, did the programme represent
good value for money?

22. www.wateraid.org
Efficiency
• What we expect: How far funding, personnel, regulatory,
administrative, time, other resources and procedures
contributed to or hindered the achievement of outputs.
Evaluation Questions:
1. Are there obvious links between significant expenditures and
key programme outputs? How well did the partnership and
management arrangements work and how did they develop
over time?
2. How well did the financial systems work?
3. Were the risks properly identified and well managed?
4. For advice on measuring value for money in governance
programmes see DFID’s Briefing Note (July 2011)
Indicators and VFM in Governance Programming, available
at: www.dfid.gov.uk

23. www.wateraid.org
Sustainability
• What we expect: Potential for the
continuation of the impact achieved and of
the delivery mechanisms following the
withdrawal of existing funding.
Evaluation Questions:
1.What are the prospects for the benefits of the
programme being sustained after the funding
stops? Did this match the intentions?
2.How have collaboration, networking and
influencing of opinion support sustainability?

24. www.wateraid.org
Innovation & Replicability
• What we expect: How replicable is the
process that introduced the changes/impact?
Refer especially to innovative aspects which
are replicable.
Evaluation Questions:
1.What aspects of the programme are
replicable elsewhere?
2.Under what circumstances and/or in what
contexts would the programme be replicable?

25. www.wateraid.org
Expected impact and change
• What we expect: Details of the broader economic, social, and political
consequences of the programme and how it contributed to the overall
objectives of the Governance and Transparency Fund (increased capability,
accountability and responsiveness) and to poverty reduction.
Evaluation Questions:
1. It is critical to demonstrate the progress in relation to the indicators included in the
GTF programme logframe. The focus is on accountability for the impact.
2. What was the programme’s overall impact and how does this compare with what was
expected? Please demonstrate from an accountability perspective if the perceived
impact was achieved and if not, why not.
3. Did the programme address the intended target group and what was the actual
coverage? Again from an accountability perspective, was the coverage reached? If
not, why not, if yes, how?
4. Who were the direct and indirect/wider beneficiaries of the programme? Again, the
importance here is to set out who these were for accountability purposes.
5. What difference has been made to the lives of those involved in the programme?
Describe the impact.
6. As you are aware, the Consultant is also conducting more detailed critical analysis on
Impact for learning purposes.

26. www.wateraid.org
Vertical Logic of Programme
Impact is the higher level
situation that the project
contributes towards
achieving
Outcome identifies
what will change and
who benefits during
the lifetime of the
project
Outputs are specific
deliverables
Human Resource and
financial inputs
LEARNING:
For the GTF Global
Consultants this
requires evidence of :
‘so what’?
LEARNING:
For the GTF Global
Consultants this
requires evidence of :
‘so what’?
ACCOUNTABILITY
For the GTF Global
Consultants this requires
evidence against
programme specific
objectives
ACCOUNTABILITY
For the GTF Global
Consultants this requires
evidence against
programme specific
objectives

27. www.wateraid.org
The Impact Assessment
Component

28. www.wateraid.org
Why conduct the impact assessment
component?
• To learn and improve:
• To enable Country programme staff, stakeholders in
country, Wateraid staff and others to really understand
what changed as a result of the programme and to apply
this to future plans
• To test and refine our understanding of how change
happens and how successful we have been in supporting
positive changes for our stakeholders:
• To what extent did we work with the right people? In the right way? How did
this all link up ?
• To what extent did the changes we expected to see along the way support
the long term changes we were aiming to influence?
• What does this tell us about the way we think we can influence change?
• What should we do differently next time?

29. www.wateraid.org
The Impact Assessment
– Focus on the “so what question”
• what’s changed?
• For whom?
• How significant/lasting are these changes for
different stakeholder groups?
• In what ways did the programme contribute
– Expect the unexpected - we are looking for
evidence of positive/negative/ intended and
unintended changes,
– Prioritise analysis over gathering information
– need for open and probing questions

30. www.wateraid.org
Key questions for the
Impact Assessment

31. www.wateraid.org
Background and context – what we need
to know (Country Programme Theory of
Change):
• The local and national context, including key social,
political and environmental conditions and how they
have changed over the life time of the programme
• Key issues that the programme planned to address
• The target groups who would ultimately benefit from
the programme and how each would benefit?
• The process or sequence of changes that would lead
to the desired long-term goal
• The assumptions that the programme made about the
anticipated process of change
• The other actors/factors who had the potential to
influence the changes sought, both positively or
negatively.

32. www.wateraid.org
Four Domains of Change
1. Changes in the ways in which CSOs function and
network, and their capacity to influence the design,
implementation and evaluation of effective WASH policies at
all levels
2. Changes in the ways that CSOs, including those
representing marginalised groups, are able to engage in
decision-making processes affecting the WASH sector.
3. Changes in the ways in which members of local
communities demand accountability and responsiveness
from governments and service providers in the WASH sector
4. Changes in the ways that Governments and service
providers are accountable to citizens and end users in the
WASH sector

33. www.wateraid.org
Each Domain is broken down further into
“areas of enquiry”
These are the key questions you need to explore across all Domains:
1. What has actually changed for each of the different stakeholder groups,
especially the poorest and most marginalized communities in relation to
WASH (positive, negative, intended and/or unintended changes)
2. How significant and/or sustainable are these changes for the different
target groups?
3. To what extent do these changes compare with baselines and changes
that were planned and expected?
4. How do they link together and/or influence each other?
5. To what extent did the GTF programme contribute to these changes?
How?
6. Who or what else might have contributed to these changes? How?
7. How confident are you in these findings (levels of evidence)?

34. www.wateraid.org
Areas of Enquiry Domain 1
Domain 1 Key Areas of Enquiry
Changes in the ways in which
CSOs function and network,
and their capacity to influence
the design, implementation
and evaluation of effective
WASH policies at all levels
• Ways in which networks have developed
and function over time
• Shifts in CSO capacity
• How this capacity change has influenced
policy and practice at
o local levels
o National level
Note: we will provide further guidelines on how to assess these areas of enquiry in
the next week

35. www.wateraid.org
Areas of Enquiry Domain 2
• P
Domain 2 Key Areas of Enquiry
Changes in the ways that
CSOs, including those
representing marginalized
groups, are able to engage
in decision-making
processes affecting the
WASH sector.
• Shift in awareness, knowledge and confidence of
marginalized groups
• Shifts in the ways that people have been able to
demand their rights
• The extent to which the voices of marginalized
people are making a difference to policy and practice
• Ways in which different CSO strategies have
influenced change (e.g. budget tracking, participation
in stakeholder reviews, etc…)
Note: we will provide further guidelines on how to assess these areas of
enquiry in the next week

36. www.wateraid.org
Areas of Enquiry Domain 3
Domain 3 Key Areas of Enquiry
Changes in the ways in which
members of local communities
demand accountability and
responsiveness from governments
and service providers in the WASH
sector
• Levels of awareness of rights in local
communities
• Ways in which media coverage
supports understanding of rights
• Ways in which citizens are influencing
policy and practice over time
• Changes in community access to WASH
• Changes in community influence over
natural resources
Note: we will provide further guidelines on how to assess these areas of enquiry in
the next week

37. www.wateraid.org
Areas of Enquiry Domain 4
Domain 4 Key Areas of Enquiry
Changes in the ways that
Governments and service providers
are accountable to citizens and end
users in the WASH sector
• Changing levels of governance,
transparency and compliance
• Changes in policy and regulation (e.g.
new policies, laws, standards, political
and institutional framework) – and
the consequences of these
• Changes in practice relating to WASH
(e.g. delivery of new services and
systems) and the consequences
Note: we will provide further guidelines on how to assess these areas of enquiry in the
next week

38. www.wateraid.org
Methodology for both components
Restrict yourselves to using a few tried and tested
tools. We suggest:
– Facilitated self assessment: building on MTR which will
support the evaluation component
• How to do this and who should be involved
• Note: we will be adding some more change questions this time
– Follow up workshop to validate findings and focus on the
impact assessment element
• How to do this and who should be involved
– Other in depths interviews /FGD with key informants as
required
• This might enable a deeper understanding of e.g. how changes
affected particular target groups

39. www.wateraid.org
Guiding Principles for Methodology
• Create an atmosphere where informants
feel able to be honest and provide critical
feedback. Use an appreciative enquiry
approach
• Ensure a mix of both qualitative and
quantitative data is gathered
• For the impact assessment – ask open
and probing questions for a deeper
understanding of change
• Findings must be backed up with evidence
and be set against the original baseline

40. www.wateraid.org
Consideration of sample size
Questions:
• How many people to interview?
• What is a “good enough sample?”
Answers:
• Be pragmatic (you have limited time but need to be
representative).
• Plan with in country focal point:
– Include people/groups/interventions which represent
• good/strong
• Medium
• Poor/weak
• Explain your sampling decisions in the methodology
section of the report ( with an indication of the level of
rigour you believe this provides)

41. www.wateraid.org
The Report
We will provide more detailed guidance over the next week but this is
the guide
35 -40 pages to include:
• Executive Summaries x 2 - (4 pages in total)
• Contents + Abbreviations (1 page)
• Methodology + challenges and limitations (2 pages )
• Country Context and introduction to the programme (3 pages )
• Evaluation Report (10 pages) – findings and conclusions under the
following headings:
– Relevance, Effectiveness, Partnership, Advocacy Equity, Value for
Money, Efficiency, Sustainability, Innovation and Replicability, Expected
Impact and change.
• Impact Assessment Report (10 pages) – findings and conclusions under the
following headings:
– Changes under each Domain
– Overall analysis of impact for different target groups
– What difference the programme has made overall
• Overall conclusions and learning for Country Programmes, for the sector
and globally (4 pages)
• Annexes

42. www.wateraid.org
Other ways to present findings
• Opportunity for Wateraid to share findings and
learning with a wide group of stakeholders -
• Target groups:
– Country programme staff
– Networks in country
– Partners
– Wateraid donors
– Sector specialists
• Supplementary ways of presenting findings (optional)
– Case studies
– Video footage
– Photos

43. www.wateraid.org
Next steps
• Maureen and Catherine to send more
detailed guidelines by Friday April 26th
• In country evaluation team to take stock of
the outcomes of the webinar and
• Prepare and send proposal to Catherine and
Maureen copied to Marta and Papa by
Friday April 26th
with a brief overview of your
plan including:
– Time line
– Your methodology
– Key informants
– Sample size and rationale for this

44. www.wateraid.org
Support and assistance
• Guidance notes to follow:
– Some thematic guidance
– Self-assessment format to use
– Learning questions
– Report format
• Online forum: you will be able to post questions, debate issues and
findings with other consultants involved in the GTF final evaluation and
impact assessment.
– Please expect an email in your inbox with a password and user details from
either Catherine or her colleague Erica Packington Erica will manage access to
the private forum.
– Please action the email immediately to guarantee access.
• E mail contacts:
– Technical advice
• Catherine Currie catherine@iodparc.com
• Maureen O’Flynn maureen@oflynn.demon.co.uk
– Logistics:
• Papa Diouf PapaDiouf@wateraid.org
• Marta Barcelo MartaBarcelo@wateraid.org
Please copy everything to Marta

45. www.wateraid.org
Questions?