By Mahjabeen Ahmed, BRAC, Bangladesh. Prepared for the Monitoring sustainable WASH service delivery symposium, Addis Ababa, Ethiopia, 9-11 April 2013.

1. Participatory performance monitoring of WASH
services at scale in BRAC WASH Programme
1
Monitoring Sustainable WASH Service Delivery
Addis Ababa, Ethiopia
9-11 April, 2013
Mahjabeen Ahmed
BRAC

2. An Introduction to BRAC
BRAC Coverage in Bangladesh:
Districts 64
Field Offices 2661
Pop. Covered 113 Million
2
BRAC‟s presence in the world

3. BRAC WASH Programme
The Programme started in 2006 with 150 sub-districts,
and has reached 248 sub-districts till date.
Components Target
(in million people)
Achievements
till Dec 2012
(in million people)
Water 2.5 1.97
Sanitation 35 27.81
Hygiene 51 42
3

4. Key Programme Strategies
• Creating demands
• Establishing Village WASH Committees
• Strong interpersonal communication component to change
behaviours
• Tailored support to ensure that hardcore poor (grants) and
poor (soft loan) are reached.
• To meet the demand supporting Rural Sanitation Centres
(loans & orientation)
• Stimulate innovation through action research programme
4

5. Monitoring Methodology
• WASH I had MIS to monitor inputs and outputs
• Inputs: e.g., number of visits, trainings
• Outputs: e.g., no. of VWCs established, no. of toilets built, of
different types
• WASH II also needed performance monitoring: that is, how
well toilets are used; how well VWCs continue to perform; to
what extent women are integrated in planning and
management; etc.
Now: MIS + QIS (quantified Qualitative Information System)
5

6. History & Rationale
• Quantified qualitative assessment methodology
• First developed by IRC and WSP in 1998
• Aim: to replace surveys, because they are extractive and
inform only central management and donors, not the users,
the VWCs and field workers
• QIS:
1. visualises to all participants where they perform well and
where they can improve („climb the ladder‟)
2. produces statistics that inform management and donors on
progress of the whole programme
3. allows to compare results over time and between locations
6

7. Scoring methodology
• QIS uses Likert scales: Participants score on a scale from
1-5, in which 1 is lowest and 5 is highest.
• Two differences:
 Each scale consist of “progressive mini-scenarios”
 Each scale starts at 0, not at 1
• Participants can see their level and can climb from 0
(“nothing to show”) to 4 (“the ideal of 4 key measurable
criteria”). The scores can be analysed statistically.
• A scale consists of “no x, x+1, x+1+1, x+1+1+1 and
x+1+1+1+1” in which each 1 is a criterion for the indicator.
7

8. Example of a QIS scale
8
Indicator 1: PERFORMANCE OF VWC SCORE
IDEAL: (1) Committee (male and female members) meets every 2
months + (2) maintains list of decisions and meeting minutes + (3)
identifies gaps and takes action + (4) mobilizes ADP funds for hard core
poor
4
(1) Committee (male and female members) meets every 2 months + (2)
maintains list of decisions and meeting minutes + (3) identifies gaps and
takes action
3
BENCHMARK: (1) Committee (male and female members) meets every
2 months + (2) maintains list of decisions and meeting minutes
2
(1) Committee (male and female members) meets every 2 months 1
No full VWC, OR, VWC exists but does not meet 0
Reason for high/low score:

9. QIS development process (1)
• Workshop 1 (Jan 2012, 1 week) with WASH
staff from HQ, all regional program
managers(20) & IRC advisors:
 Set indicators
 Formulate & review scales
 Define terms
 Design scoring process
 Finalize scales & work plan
• Workshop 2 (March 2012, 1 week)
 First pilot training & field testing of QIS
9

10. QIS development process (2)
• Adjustment of scales, process and manual (April 2012)
• Full-scale pilot (August-Sept 2012) with 432 households
(144 UP/P/NP, 36 VWCs, 12 schools and 12 RSCs in 4
upazilas in 4 geographical zones
• Workshop 3 (Oct 2012, 1 week):
 Analysis of data & experiences and report
 Adjustment of scales and manual
 Selection of independent monitoring teams
 Sample design: sample frame, size, sampling
methods
10

11. QIS development process (3)
• Training of 30 independent monitoring teams (Dec, 2012):
1 male and 1 female in each team, 3 batches for 6 days
• 3-stage sample: (1) Random choice of 50 Sub districts from WASH I and
50 new unions from WASH II with probability proportional to size
(2) Minimally three or 3 VWCs/ sub-district
(3) 27 HHs per VWC (3x9 HH for UP/P/NP)
11
 8100 HHs
 400 schools
 300 VWCs
 300 RSCs

12. The QIS indicators for BRAC WASH
Village WASH Committee 1. Safe and protected drinking water source
2. Performance of VWC
3. Women‟s participation
Households 4. Safe and protected main drinking water source
5. Drinking water management from source to cup
6. Sanitary and hygienic household latrine
7. Who uses the latrine
8. When are latrines used
9. Hand-washing provisions after defecation
10. Sludge management when latrine pit is full
Schools 11. Sanitary and hygienic school toilets
12. Student brigade
13. Menstrual hygiene management
14. Performance of school WASH committee
Rural sanitation centers 15. Performance of RSCs
12

13. Data collection and analysis
• 1 male & 1 female staff collected data
• Data directly entered into a smart phone
• Data sent by smart phones to the QIS
data base
• No data entry persons needed in Dhaka
• ICT and WASH staff cleaned the
database
• BRAC‟s QIS manager and IRC‟s QIS
advisors did a first analysis of the data
using Epi-info open source software
13

14. Sample locations of:
 8100 HHs
 400 schools
 300 VWCs
 300 RSCs
14

15. • Scientific sampling methods
• Unique bar code avoids duplication when data is
submitted more than once, and allows re-visit if
needed
• Greater reliability through:
 No manual data entry (high error rate)
 Double data entry in smart phone and
on paper
 Team compared scores on internal consistency
before sending/uploading
15
Quality Assurance

16. Implementation and support
• The teams started the data collection just after the training
• Each team could always call Managers or ICT to solve any
problems with data collection or smart phones
• Real time data
• The hard copies were checked rigorously to detect errors & for
improvements next time
• One person was called back for retraining when consistent
data errors detected
• For gender equality, team members alternated roles of
process facilitator and data entry
16

17. Summary of preliminary findings
17

18. • All but 1 VWC are functional and
meet regularly
• 99% VWCs at or above bench
mark
• Almost 1 in 3 VWCs meet all 4
criteria (meet 1x/2mnth, record
decisions and minutes, identify
gaps and take action, mobilise
govt. funds for ultra poor)
BUT
• Almost 70% can still improve
performance further by mobilizing
government funds meant for the
ultra poor
• 26% can also improve by noting
gaps and taking actions
QIS Findings on VWC performance
PERFORMANCE OF VWC %
IDEAL: (1) M+F members meet every 2
months + (2) record decisions & minutes
+ (3) identify gaps & takes action + (4)
mobilizes ADP funds for HCP
31
(1) M+F members meet every 2 months +
(2) record decisions & minutes + (3)
identify gaps & takes action
42
BENCHMARK: (1) M+F members meet
every 2 months + (2) Record decisions
and minutes
26
(1) M+F members meet every 2 months 0
No full VWC, OR, VWC exists but does
not meet
1
TOTAL 100
18

19. Safe management of
drinking water in homes
• Above benchmark, P and
NP perform better than
HCP (43%, 52%, 35%)
• 22% HCP, 27% P and 33%
NP meet all 4 criteria on
safe drinking water from
source to cup
• Others (67% HCP, 67% P
and 64% NP) still 1-4 risky
DW management practices
5% 4% 5%
19% 19% 19%
30%
28%
21%
13%
16%
19%
22%
27% 33%
11%
5%
4%
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
HCP POOR NON-POOR
Na
level 4
level 3
level 2
level 1
level 0
BM
19

20. Hand Washing
Provisions at toilet
IDEAL: (1) Enough water to wash
hands carried or available in or near
latrine + (2) soap/soap solution in
plastic bottle at latrine + (3) water for
hand-washing is from safe source +
(4) there is a special hand-washing
station
(1) Enough water to wash hands
carried or available in or near latrine
+ (2) soap/soap solution in plastic
bottle at latrine + (3) water for hand-
washing is from safe source
BENCHMARK: (1) Enough water to
wash hands carried or available in or
near latrine + (2) soap/soap solution
in plastic bottle at latrine
(1) Enough water to wash hands
carried or available in or near latrine
(0) No provisions for hand-washing
carried or available in or near latrine.
• Presence of hand wash
facility as proxy for hand
washing behaviour.
• Majority of scores at or
above BM. HW
provisions at toilets is
still a focus area: 10%
are still at the score 0.
20

21. QIS findings from the School
21
Toilets for girls Student brigade
Menstrual
hygiene
management
Above BM 80% 68% 68%
At BM 12% 20% 5%
Below BM 8% 12% 26%
Total 100% 100% 100%

22. Experiences
• Performance monitoring concept was new: VWCs,
householders and field staff initially experienced it as
evaluation/judgemental
• Once understood, the „climbing the ladder‟ approach is
much appreciated by all
• Small details could be easily captured
• Comparing written and phoned data showed need for some
correction and is good learning for next round (Dec. 2013)
22

23. Thank you
23