Learning is a critical part of the HRITF RBF portfolio, with all programs benefiting from an embedded impact evaluation and in some cases, complemented by qualitative research components such as process evaluation studies. The presentation discusses the following topics: 1. Using RBF at the community-level to address demand side barriers This presentation elaborates on the early evidence and the rationale for using RBF at the community level. It will share lessons learned from the implementation of community RBF at country level. 2. Using RBF to Strengthen Quality of Care: Early Lessons This presentation discusses the broader policy implications of using RBF to strengthen the quality of care. It will explore how Measuring and Paying for the Quality of Care has been operationalized and will highlight the experience of Nigeria. Lastly, it will focus on measuring and Analyzing the Quality of Care from the Impact Evaluation perspective.

1. Setting a Path for Improved Health Outcomes
Results-Based Financing: the Evidence thus Far

2. Early evidence on Results-Based Financing:
Demand and Community Based incentives

3. Evidence from a preliminary analysis of
financial incentives for health
3
 Financial incentives have worked, but…
– Demand- and supply-side incentives work on different
margins. Demand-side incentives encourage people to go to
a facility, while supply-side incentives encourage health
providers to deliver more and better care to people who have
made it to the facility
– Demand- and supply-side incentives are complements, and
are best combined;
– Community-based incentives, for example incentives to
community health workers, could serve as “bridge” between
supply and demand.
– But few evaluations so far have looked at the combination of
supply and demand side incentives and at the role of
community-based incentives.
– We need to learn more.

4. Conditional cash transfers and children
health outcomes
 Some health outcomes and behaviors might be easier to
influence from the demand side (patients, population) rather
than from the supply side (health care providers).
 See example from Rwanda
 (Conditional) cash transfers have been widely used and
evaluated as a social protection mechanism.
 When they are conditional, the conditions are linked to
educational and/or health behaviors.
 They usually have impacts on reducing poverty, but also on
improving education and health outcomes.
4

5. 5
Gender and Conditionality: A Randomized
Evaluation of Alternative Cash Transfer Delivery
Mechanisms in Rural Burkina Faso

6. Cash Transfer Pilot Program
Randomization Plan
6
75 villages
(2775
households)
_________
|
_____________
|
_
|
______|_____
|
_____________
|
________
|
15 villages
(540
households)
Randomized
CCT to
Father
15 villages
(540
households)
Randomized
CCT to
Mother
15 villages
(540
households)
Randomized
UCT to
Father
15 villages
(540
households)
Randomized
UCT to
Mother
15 villages
(615
households)
Randomized
to Control
Group

7. Cash Transfers Overview
 Transfer amount:
– Ages 0-6: 4000 FCFA/year
– Ages 7-10 (Grades 1-4): 8000 FCFA/year
– Ages 11-15 (Grades 5+): 16000 FCFA/year
 $1 USD = 500 FCFA
 CCT:
– Ages 0-6: Quarterly visits to health clinic for preventive
care (growth monitoring)
– Ages 7-15: School attendance rate>90%
 UCT:
– No requirements
7

8. Research Summary
 Consider broad measure of welfare outcomes: education,
health, livestock, agriculture, demographics,
assets/infrastructure
 For child education and health outcomes, conditional cash
transfers outperform unconditional transfers
 Giving cash to mothers does not lead to significantly better
child education or health outcomes
 Evidence that giving cash to fathers improves child health in
bad rainfall years
 Cash transfers to fathers yields more investment in livestock,
cash crops, and improved housing
8

9. CCT and adolescent health outcomes
including HIV prevention
 Traditionally CCTs target education outcomes as
well as mother/child health outcomes.
 More recently they have also been tested as a way
to influence adolescent/young adults health
outcomes and behaviors, in particular for HIV
prevention.
 Those are behaviors and outcomes which are likely
to be difficult to influence through a classic supply-
side RBF program.
9

10. 10
STIs?
HIV?
Baird, Garfein, McIntosh and Özler, 2012

11. 11
STDs?
HIV?
+
STIs?
HIV?

12. 12
Study
population
(N=1,328)
Control
(N=827)
Treatment
(N=501)
Unconditional
Cash Transfer
(N=265)
Conditional
Cash Transfer
(N=236)

13. 13
Study
population
(N=1,328)
Control
(N=827)
Treatment
(N=501)
Unconditional
Cash Transfer
(N=265)
Conditional
Cash Transfer
(N=236)
Relative risk (compared to
control, adjusted)
Pregnant now : 0.16 (p<0.05)
Partner≥25 : 0.36
HIV : 0.47
HSV-2: 0.08 (p<0.05)
Relative risk (compared to
control, adjusted)
Pregnant now : 1.17
Partner≥25 : 0.08 (p<0.05)
HIV : 0.29 (p<0.05)
HSV-2: 0.37
NB UCT significantly different than CCT only for “pregnant now”
outcome

14. Impact Evaluation of the Rwanda
Community Performance-Based
Financing Program
College of Medicine and Health
Sciences School of Public Health

15. Background: Community PBF (Second
Generation)
15
Since 2009, Community Health Workers (CHWs) were
paid for reporting on health indicators in their
communities
Additional components were added through the
Community Performance-Based Financing Program in
order to promote targeted services
This study evaluates the impact of 2 interventions that
were added to the scheme:
1. Performance incentives for CHW cooperatives
2. Demand-side in-kind incentives

16. Background: organization of CHWs in
Rwanda
16
Each village has 3 volunteers serving as Community Health
Workers (CHWs).
Multidisciplinary
CHWs
CHW in Charge of
Maternal and
Neonatal Health
Criteria
• Can read and
write
• Age 20-50
• Lives in the
village
• Elected by the
village
residents

17. Background: organization of CHWs in
Rwanda
17
All the CHWs within the catchment area of a health center are
organized in a CHW cooperative.
Cooperative

18. Background: organization of CHWs in
Rwanda
18
70% of payments received by a cooperative must be invested in
income generating activities (IGAs).
30% of the payments and revenues from the IGAs are given to
cooperative members. It is up to the cooperatives to determine
distribution rules.

19. Intervention #1: Performance Incentives for
CHW Cooperatives
CHW cooperatives received financial rewards for:
1. Nutrition monitoring: # children 6-59 months monitored
2. Timely Antenatal Care: # of women accompanied/referred
within first 4 months of pregnancy
3. In-Facility Delivery: # of women accompanied/referred for
assisted delivery
4. New Family Planning users: # referred to health center
5. Regular Family Planning Users: # regular users at health
center
 4 indicators related to TB and HIV were added at a later
stage and not evaluated
19

20. Intervention #2: Demand-Side In-Kind
Transfers
20
Women received gifts for seeking care for the following services:
* Women can only receive the gifts for one pregnancy every 3
years.
Eligibility* Value (Ceiling) Suggested Package
Initiation of Antenatal Care
during first 4 months of
pregnancy
5 USD Adult cloth and water treatment
tablets OR baby cloth package
and water treatment tablets
Delivery in health center 6.67 USD Baby soap, baby shawl and baby
bed sheets
Initiation of Postnatal Care
during the 10 days after
delivery
3.33 USD An umbrella and water treatment
tablets OR Adult cloths

21. Research Questions
1. Do the demand-side in-kind transfers and the performance
incentives to CHW coops increase
– Initiation of prenatal care within first 4 months of
pregnancies?
– Total prenatal care visit?
– In-facility deliveries?
– Rate of postnatal care within 10 days after delivery?
2. Is there a multiplicative effect when both interventions are
implemented?
3. Do the performance incentives to CHW coops affect
– Behavior and motivation of the CHWs?
– Use of modern contraceptives?
– Growth monitoring of children under 5
21

22. Study Design: RCT
22
198 sectors (sub-districts) were randomly allocated into 4
study arms:
* Coops paid for reporting received the average amount
received by the coops paid for performance
Payments to CHW Coops
For Reporting* For Performance
Demand-Side Transfers
No C S
Yes D D+S

23. Timeline
23
2010 2011 2012 2013 2014
February-
May 2010
• Baseline
Survey
November 2013-
June 2014
• Follow-up Survey
October 2010
• Interventions
Introduced
February 2013
• Last transfer
of funds for
in-kind
transfers

24. Results: Maternal Health Services
 Indicators:
– Timely ANC
– In-facility deliveries
– Timely PNC
 Sample of women with most recent birth in their
village
– Pregnancies resulting in a live birth
24

25. Results: ANC visit within first 4 months of
pregnancy
25
50%
55%
60%
65%
70%
75%
80%
85%
Control Demand Supply D + S
Timely ANC 72% 82% 74% 80%
Timely ANC
• A positive and significant (at the 1% level) impact of the demand-side in-kind
incentives of about 10 percentage points
• The CHW incentives are not found to have a significant effect
• No difference between the ‘Demand’ and the ‘Demand+Supply’ treatment arms

26. Results: at least 4 ANC visits
26
25%
30%
35%
40%
45%
50%
Control Demand Supply D + S
4 ANC 40% 46% 43% 45%
Four or more ANC visits
• Not targeted by the program!
• Higher in the intervention sectors, but not statistically significant at the 10%
level

27. Results: Skilled-attended in-facility delivery
27
70%
75%
80%
85%
90%
95%
100%
Control Demand Supply D + S
Deliv 94% 95% 96% 94%
In-Facility Delivery
• No statistically significant difference between the treatment arms
• Rate has increased substantially in the duration of the study for other
reasons

28. Results: PNC within 10 days after delivery
28
0%
5%
10%
15%
20%
25%
Control Demand Supply D + S
Timely PNC 13% 22% 11% 20%
PNC within 10 days after delivery
• A positive and significant (at the 5% level) impact of the demand-side in-kind
incentives of about 7 percentage points
• Not targeted by the CHW incentives intervention

29. Key Findings: Demand-Side In-Kind
Incentives
• The demand-side in-kind incentives caused an
increase in timely ANC and PNC services
• Although some challenges in procurement and
frequent stock outs
• Although some health centers independently
implemented their own demand-side incentives
strategies to promote utilization
• Although funding ended before end-line data
collection
• Consistent with findings in other countries that
implemented demand-side cash transfers
29

30. Key Findings: Performance Incentives to
CHW Coops
• No impact of incentives to CHW cooperative on
targeted indicators, CHW behaviors and CHW
motivation.
• Potential reasons for lack of impact
– Incentives were too low
– Collective reward but individual effort
– Pay-for-reporting could have already oriented the
CHWs towards targeted indicators
– Limited scope given the many supply-side
programs targeting the same indicators
30

31. Research Team
 Ministry of Health
– Fidel Ngabo
– Cathy Mugeni
 University of Rwanda
– Ina R. Kalisa
– James Humuza
– Jeanine Condo
– Vedaste Ndahindwa
 The World Bank
– Gil Shapira
– Netsanet W. Workie
– Jeanette Walldorf
31
The study was funded by the Health Results
Innovation Trust Fund (HRITF)

32. The Case of Community RBF

33. What is cRBF?
 Community RBF: a set of different practices:
– Based on the idea of contracting (cRBF)
– Separation of functions (purchaser, provider,
regulator and verifier)
RBF is: “a cash payment or non-monetary transfer
made to a national or subnational government,
manager, provider, payer or consumer of health
services after predefined results have been
attained and verified. Payment is conditional on
measurable actions being undertaken” (Musgrove,
2010)
33

34. Rationale for cRBF?
34
What is the objective?
Provide services at the most peripheral and decentralized
level
Contracting of CHW
Often attached to a health facility
Stimulate the demand side
Awareness meetings
Contacts with the population
Vouchers and incentives
Achieve health related behavioral changes
Part of all cRBF, sometimes stated more clearly (The Gambia
and Congo)
Health promotion /
awareness [HP]
Use of services [US] Health outcomes [HO]

35. Who is contracted in cRBF?
 Who are the community actors contracted in cRBF?
– Community Health Workers: in charge of providing specific
services, often preventative care and awareness campaigns [in
the spirit of the 1977 Alma-Ata conference]
– Health Facility Committee members: co-managers of the
health facilities, intermediaries between population and service
providers [in the spirit of the 1988 Bamako conference]
– Traditional healers –a large variety: traditional midwifes,
herbalists, etc.
– Other community actors:
▫ Village committee
▫ Community-based organizations
 What is not included under cRBF?
– Individuals directly: then closer to Conditional Cash Transfers
(CCTs)
35

36. Lessons Learned from cRBF Operations

37. Country cRBF experiences
 Contracting of Community (Health) Workers:
– Benin
– Cameroon
– Republic of Congo
– Rwanda
 Contracting of Health Facility Cie.
– DR Congo
 Contracting community organizations
The Gambia
 Demand-side and voucher schemes
(not discussed here)
– The Gambia
– Rwanda
– Congo
37

38. Lesson 1:
cRBF programs should be designed taking into
consideration contextual factors
38

39. cRBF programs should be designed taking into
consideration contextual factors (Cameroon)
Example of Cameroon:
 PBF Indicators started improving in HF but stagnated despite much
efforts by health facilities
 Reports of many drop outs concerning vaccinations, post natal
consultations antenatal consultations and use of family planning
among women.
 Nutritional concerns of children were poorly addressed by program
 Therefore something had to be done to re-stimulate demand for health services
by the community
 Reflection of the Government and partners led to identifying a Community PBF
approach as a strategy worth trying
 Experience of some health facilities sub-contracting with Health Committee
Members had proven it’s worth in referrals and search for drop outs
 Need therefore to contract Community Health Workers in a formal manner
 a cPBF pilot was then started in July 2015 with a Community Monitoring
component to strengthen the voice of the community in health care delivery
39

40. Lesson 1: Experience from RoC
 Each Context is unique
– Avoid Copy and paste
 Context is essential to define the CPBF Model of
RoC
– Low coverage for some indicators
– Absence of community networks
 Objective: support households in the health seeking
behaviors.
 Interventions:
– Put in place the community relays
– Action plan signed with the household
40

41. Lesson 2:
Existing community structures should be
assessed and, where possible, strengthened
using cRBF
41

42. Contracting Community Committees: The
Gambia
 Most communities in The Gambia have:
– Village Development Committees (VDC) responsible for all
development activities of the community; and
▫ Village Support Groups (VSG) comprising 4 women and 2 men who,
with the VHW and TBA, are trained to promote optimal maternal, infant
and young child feeding practices. They are an arm of the VDC.
 During the design stage of the Maternal and Child Nutrition and Health
Results Project, anchored on PHC, it was unanimously agreed that the VDC
be contracted to implement the Demand side of the Project
 This was strengthened by the type of indicators which could not be
contracted to individuals: the demand side (cRBF) indicators focused on
knowledge and practice
 The verification of these indicators is done using a survey (LQAS) –
therefore the entire community is contracted through the VDC
 20% of the quarterly subsidy payment is given to the VSG as an incentive
while the balance goes into the implementation of a community
development project identified through a PRA
42

43. Experience of Benin (Similar to Cameroon)
 Preexistence of community health workers:
sensitize the population on health, refer patients to
the health center
 But fragmentation of package of services
depending on sources of funds
 cRBF relies on existing CHWs and train them on
the complete package
 Then, sub-contract between individuals and HF
 The Health center: Coordination center to share
good practices, to declare results, group monitoring,
supervision and payment
43

44. Lesson 3:
Broad participation in the selection of indicators
is key…
44

45. Experience of Benin
 Involve all actors in the process to prioritize
indicators:
– Central level MoH, Vertical programs, Donors,
district level, local levels,…… (with focal persons
at all steps)
 Build ownership :
– Good understanding by all stakeholders
– Appropriate indicators for the implementation of
the PIHI program
– Coordinate and prioritize (All indicators cannot be
part of the package)
45

46. Lesson 4:
But it’s important to limit the number of indicators
to ensure feasibility and quality
46

47. Variety and scope of indicators varies
 Indicators can be at all levels
 The number of indicators matters:
– The Gambia (9): Health promotion
– Benin (9): Referral system
– DRC: Hybrid (functionality indicators, health
promotion)
– Cameroun (20): Referral system, service
utilization
– RoC: Health promotion
47

48. RoC: Advantages with few indicators
 Better verification:
– Good quality data: reliable
 Better analysis of data collected:
– Areas of weaknesses and strengths
 Low cost of transactions for verification, high
cost for individual indicators (Motivating for
CHWs)
48

49. Lesson 5:
Data collection tools should be simple
49

50. Make management tools simple
50
Current Challenges
1. Tools for community health workers and other community
members are too complex
2. Tools are not effectively used because they are time
consuming
3. Tools and processes are designed for the purchaser or the
regulator rather than the users and community
Recommendations
1. At the community level, tools should be simple and easy to use
2. Tools should be validated by the relevant community actors
3. Strengthen the community capacity for monitoring

51. Lesson 6:
Systems are needed to monitor and maintain the
quality of training at all levels of the health
pyramid
51

52. Systems are needed to monitor and maintain the
quality of training at all levels of the health pyramid
52
Current Challenges:
1. To decentralize, there is a need of training in cascade mode.
2. But, the cascade mode doesn’t ensure the quality of training
at peripheral level A (100%)-- B (85%)--C ( 70%)-- D ( 45%)
3. The content of the training is losing some key information
4. During the implementation, new issues arise
5. Differentiated adaptation
Recommendations:
1. Ensure quality of training at lower levels
2. More supervision and monitoring of the trained community actors
3. Benchmark the good practices of those who succeed to support the
weak CHWs

53. Lesson 7
Payments to CHWs should be timely
53

54. Payments to CHWs should be timely
(Cameroon)
- During pilot period for cPBF payments from central level to health facilities
were often delayed. At first facilities waited for PBF subsidies to arrive
before paying CHWs, this led to long delays in paying CHWs, leading to
demotivation and frustration of CHWs
- To improve on the retention of the CHWs, the payment model was
revised.
- Now the quarterly facility contracts stipulate that the health facility should
pay the CHWs monthly as soon as their verification is done; using facility
resources (mix of cost recovery, PBF subsidies, etc.).
- Difficult to convince all facilities to accept this approach, but by including it
in the facility contract they, CVA was able to negotiate this payment
mechanism.
- After several months facilities have noted that it is possible to ensure
timely payment of CHWs
- CHW motivation and retention has improved. Model scaled up to other
regions.
54

55. Lesson 8:
ICT can be very useful but it should be built on
solid systems and carefully tested
55

56. Use of Mobile Devices for Data Reporting and
Verification: The Gambia Experience
 The Gambia started with strengthening the already existing
HMIS and incorporating RBF indicators
– Data collection and reporting tools were reviewed and
updated
– The DHIS2 database updated to reflect the new information
– PHC Circuits were re-demarcated to fit within health facility
catchment areas
 The country team is now considering the gradual introduction
of the use of mobile devices starting with verification using
tablets
 Also considering the use of mobile money for the payment of
CCTs to pregnant women
56

57. Lesson 9:
There’s still a lot to learn!
57

58. Learning Opportunities
 How best can community level data be used to inform
activities?
 How to ensure that CHWs only provide the services
they are meant to provide?
 How to appropriately share data with communities
and promote community ownership of activities?
 What is the impact of sub-projects funded through
community incentives?
 Why was there high CHW drop-out after initial
training?
 How best to do verification of community data?
58

59. What are we learning?
Projects in the World Bank’s current portfolio of cRBF
are in the process of answering some outstanding
questions.
 RoC and DRC are evaluating a strategy of paying
health centers to conduct home visits jointly with
community agents
 Cameroon is assessing the impact on uptake of
services of health centers subcontracting
community health workers
 In the Gambia, the impact on health behaviours and
uptake of health services of performance payments
to community organizations is being assessed
59

60. THANK YOU

61. Results-Based Financing & Quality of Care:
Measuring and Paying for Quality Improvement

62. Session Outline:
Measuring and Paying for Quality
I. Existing Instruments and Methods
II. Using Data for decision making
III. Verifying Data Accuracy
IV. Innovations in Measuring and Paying for
Quality
62

63. 1. Existing Instruments and Methods
63

64. Measuring if the right inputs are in place
64

65. Liberia: Quality Assessment/
Monitoring Tools
1
Complicated and assisted delivery
(including C-section)
Any labor that is made more difficult or complex by a deviation from the normal
procedure. Complicated delivery is defined as: assisted vaginal deliveries (vacuum
extraction or forceps), C-section, episiotomy and other procedures.
17
2 Normal deliveries of at risk referrals
High-risk pregnant women referred by health center to the hospital but delivered
normally. A high-risk pregnancy is defined as: evidence of edema, mal presentation,
increased BP, multi-parity, etc.
17
3
Counter referral slips returned to health
facilities
Hospital returns counter referrals letter with feedback on the referred patient to the
referring health center. The counter referral letter is completed in triplicate, with one
also given to the patient, and one retained by the hospital.
2.5
4
Newborn referred for emergency
neonatal care treatment and treated
Newborns referred for emergency neonatal care due to: perinatal complications, low
birth weight, congenital malformation, asphyxia, etc.
5
6
Referred infants and under-fives with
fever
Any surgical procedure that does not involve anesthesia or respiratory assistance. 2.5
7 Minor surgical intervention
Any surgery in which the patient must be put under general spinal/anesthesia and
given respiratory assistance. Major surgery in the case of this package of services is
defined as any of the following: Herniarraphy, Appendectomy, Myomectomy,
Sleenectomy, Salpingectomy, Hysterectomy, Thyrodectomy, Mastectomy.
5
8
Major surgery (excluding CS, including
major trauma)
Patients transferred from a lower-level facility (health center or health clinic) to the
hospital for emergency treatment.
18
9 Patients transported by ambulance 2.5
10
Number of training sessions held by
faculty for nurses, midwifes and PA
according to in-service curriculum and
defined protocols.
These indicators will incentivize the in-service training activities. 50
11
Number of nurses, midwifes and PAs
that received specialized in-service
training, relevant to benchmarks
10
Verified
Total
EarningsDefinition
Six Hospitals Total
Fee (USD)Indicators Claimed
(c) Quantity Checklist
Actual % Earned Points
1. Obstructed Labor 0.80 3.87 100% 33% 1.29
2. Hemorrhage 1.00 4.84 100% 71% 3.45
3. Maternal Sepsis 1.00 4.84 100% 50% 2.42
4. Eclampsia 0.70 3.39 100% 47% 1.59
5. Neonatal Asphyxia 1.00 4.84 100% 67% 3.23
6. Neonatal Sepsis 1.00 4.84 100% 54% 2.61
7. Prematurity 0.50 2.42 100% 47% 1.14
8. Maternal Newborn Best Practices 1.00 4.84 100% 54% 2.61
9. ETAT 1.00 4.84 100% 33% 1.61
10. Malaria 1.00 4.84 100% 71% 3.45
11. Pneumonia 1.00 4.84 100% 50% 2.42
12. Acute Diarrhea 0.80 3.87 100% 47% 1.82
13. Severe Acute Malnutrition 0.60 2.90 100% 67% 1.94
14. Surgical Safety 1.00 4.84 100% 54% 2.61
100% 60.00 100% 53% 32.20Total/Average
Childbirth:
Maternal-Newborn
Pediatric
(in-patient care)
Surgical Care
Quarter I
III. Process of Care
Detailed Score
Checklists
Weight (by
importance)
Point
Allocation
Max %
(b) Process of Care
Quality Checklists
Score
1.GENERAL MANAGEMENT (30pt)
2. HUMAN RESOURCES FOR HEALTH (16pt)
3. HYGIENE AND MEDICAL WASTE DISPOSAL (27pt)
4. DRUGS MANAGEMENT (30 pt)
5. EQUIPMENT AND SUPPLIES (84pt)
TOTAL %
Date of Verfication
TOTAL (187pt)
REPUBLIC OF LIBERIA
Ministry of Health and Social Welfare (MOHSW)
Hospital Quarterly Quality Assessment
Name of the Hospital
Name of Team Leader of Quality Verification
Verification Period
Quarterly Quality Verification Score
I. Management
II. Structural
(a) Management and
Structural Checklist
Indicators
Max Points
Actual Points
Quarter I
1. General Management 30 2.6
2. Human Resources for Health 16 9
3. Hygiene and Medical Waste Disposal 27 0
4. Drugs Management 30 8
5. Equipment and Supplies 84 48
6. Aggregated Process of Care Score 60 32
Total 247 100
Total Percentage 100% 40%
Total Quality Bonuses (USD) 159,678 64,517
PBF Bonus
Calculation Tool
Business/Operation Plan
Health Worker Bonus
Allocation
LHSSP Indices Tool for Bonus Allocation to Individual Health Workers for Hospitals
1 200 50 30 300,000 0 6,944
2 200 70 30 420,000 0 9,722
3 150 80 30 360,000 0 8,333
4 - - -
5 - - -
6 - - -
7 - - -
8 - - -
9 - - -
10 - - -
11 - - -
12 - - -
Quarter:
Total PBF Incentives Earned
% for Individual Bonus
Attendance
points [C]
Hospital Name
Total Individual Bonus
Redemption Hospital
July-Sept 2013
No Name of staff
Staff
category
Monthly
salary [A]
Perfor-
mance
points [B]
$50,000
50%
$25,000
Total points =
[A] x [B] x [C]
Indices of
the period
PBF
individual
bonus
Signature of receipt
Min
50%
Max
50%
~60%
~20%
~20%
(1)Continuousmonitoring
(d) Impact Evaluation
Measuring processes and results
65

66. Liberia: Standards for Management Obstructed Labor:
Illustrative Checklist Distilling Essential care Items
(admission, labor)
Chart review elements (see chart review guide for specific
criteria) ; each element if recorded = 1 point
Charts
1. Admission 1 2 3 4 5
1. Cervical dilation recorded at admission (# of cm)
2. Contraction frequency and duration charted at admission
3. Fetal presentation charted at admission
4. Partograph started when cervical dilation 4 cm or greater
Admission Score (x/4)
2. Labor Monitoring (partograph)
1. Cervical dilation recorded at least every 4 hours
2. Frequency and duration contractions recorded at least every 30
minutes
3. Fetal HR recorded at least every 30 minutes
Labor Monitoring Score (x/3)
Each item has chart review guide
that defines criteria
Five patient charts reviewed:
average score (% adherence best
practices) links with bonus
66

67.  Record Reviews
 Simulations of routine labor
and delivery, postpartum
hemorrhage and eclampsia
using Mama Natalie
 Simulation of newborn
resuscitation using Neo
Natalie
 Simulation of surgical safety
checklist use
 Patient interviews by phone
include basic quality tracers
(access to sanitation facilities;
recall health education
messages; informal payments
and general satisfaction using
a Likert scale)
https://youtu.be/_miYvoWosS4
Kyrgyzstan: multiple approaches to measuring
quality
67

68. 2. Using Data for Decision Making
68

69. Nigeria: Institutional Deliveries increased from
20% to 44% during 2015(120% increase)
69
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Jan-15
Feb-15
Mar-15
Apr-15
May-15
Jun-15
Jul-15
Aug-15
Sep-15
Oct-15
Nov-15
Dec-15
Population Coverage for Institutional
delivery – PBF districts
National (PBF) Adamawa
Nasarawa Ondo
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Jan-15
Feb-15
Mar-15
Apr-15
May-15
Jun-15
Jul-15
Aug-15
Sep-15
Oct-15
Nov-15
Dec-15
Population Coverage for Institutional
Deliveries – DFF districts
National (DFF) Adamawa
Nasarawa Ondo

70. 70
Large variability in Institutional Deliveries across Health
Centers Fufore District, Adamawa State Nigeria during 2013
-
20
40
60
80
100
120 Pariya HC
Chigari HC
Dasin Hausa HC
Farang HC
Ribadu HC
Furore MCH HC
Choli HC
Gurin HC
Malabu HC
Karlahi HC
Wuro Bokki HC
Kabilo HC
Saint Mary's Clinic HC
Mayo-Ine HC

71. Burundi: Average total quality score for health
centers, by province and time
71
0.0
20.0
40.0
60.0
80.0
100.0 Mwaro
Muramvya
Kirundo
Cibitoke
Buja-Rural
Kayanza
Ngozi
Makamba
Rutana
Bubanza
Bururi
Gitega
Karuzi
Muyinga
Ruyigi
Cankuzo
Buja-Mairie

72. Quthing District: average quality in health centers is the
same after 12 months piloting of PBF due to autonomy
problems
72
0
10
20
30
40
50
60
70
80
90
100
General_Management
Child_Survival
Environmental Health
General_Consultations
Reproductive_Health
Essential_Drug_Management
Tracer_Drugs
Maternal_Health
STI_HIV_TB
Comm_Based_Services
2Q14 2Q15

73. 3. Verifying Data Accuracy
73

74. NIGERIA: Quality of Care at PHCs:
Raising the Bar
0
10
20
30
40
50
60
70
80
90
100
December
March
June
September
December
March
June
September
December
March
June
September
December
March
2011 2012 2013 2014 2015
PercentageQualityScore
Adamawa
Nasarawa
Ondo
National
 Quality of care also improved significantly with emphasis on structural and process of
care indicators (higher emphasis on process end 2013 leads to drop)
 Overall patient perceptions on quality of care is relatively satisfactory
 Counter-verification of the quality: relative large discrepancies
74

75. Concordance in 2015 and 2016
75
90%
97%
81%
66%
61%
66%
95% 96%
76% 76%
85%
92%
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Fufore LGA Mayo Belwa LGA Wamba LGA Karu LGA Ile oluji / Okeigbo
LGA
Ondo East LGA
2015 Average Concordance
2016 Average Concordance
0-5% Concordance

76.  Ex-ante verification by district health team may be
too gentle and not accurate: too close for comfort or
still old fashioned ‘filling under the banana tree’?
 Regular counter-verification with credible sanctions
are an important requirement
 Specifying incentives for district supervisors also
seems a promising route (share of earnings;
accreditation status; carrots and sticks)
 Introduction of modern ICT such as tablet based
checklists, which embed meta data (location; time;
interviewer passcode) seem a promising approach
too
76
Challenges to Measuring and Rewarding Quality
Performance

77. 4. Innovations in Measuring and Paying for
Quality
77

78. Virtual Patient presents
with symptoms
Provider cares for a
variety of clinical cases
Provider goes through
the different clinical
domains as when they
see a patient
Vignettes Provide a Standard Measure of Practice
78
 Take History
 Conduct a Physical Exam
 Order Tests
 Make a provisional diagnosis
 Decide on treatment

79.  Tablets for quantified quality checklists (‘balanced
score cards’) with automated uploads to a cloud
based database and public dashboard. Offline data
entry possible
 (as above) Tablet based solution for Vignettes (under
development)
 Smart phone for community client interviews. Off line
data entry possible. Automated uploads to a cloud
based database and public dashboard. Results
impact on performance payments
 Web-based public dashboard for performance
benchmarking
79
Technology Aids for Quality Measurements
in PBF

80. 80

81. 1. Quality is poor and varied
2. Much improvements in access and
structural elements of care
3. Improving clinical processes remains the
big immediate challenge
4. Innovations are happening in the space
of measuring clinical processes
5. Data from measurement needs to
translate to decisions
In Summary
81

82. Improving Quality of Care Using
Measurement, Comparison, Validation

83. • If we can measure:
• Target performance
• Knowledge to perform
• Capacity to perform
• Performance
• Then the gap between
performance and targeted
performance can be
broken down into:
• The know gap
• The know-can gap
• The can-do gap
Target
Performance
Gap
Know Gap
Know-Can
Gap
Can-Do
Gap
Target
Knowledge to perform
Capacity to perform
Performance
Three Gap Model of Performance
(Leonard et al., 2015)
83

84. The Three Gaps in Liberia from 2013 to 2015
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
30%
39%
28%
11%
13%
25%2%
2% 2%
57%
46% 45%
performance can do gap know can gap know gap
2015 full 2015
partial
2013
partial
The three
samples include
10 hospitals in
2015 (2015 Full)
and 4 hospitals
observed in both
2013 (2013
partial) and 2015
(2015 partial)
84

85. What do we learn from these gaps?
 This is not a pure impact evaluation: the biggest
driver of changes in this data is the Ebola crises,
not the RBF.
 The biggest change from 2013 to 2015 is an
increase in the can-do gap, which suggests a drop
in motivation consistent with the crises.
 The biggest gap is clearly the knowledge gap, but
does this mean improved knowledge leads to
improved performance?
85

86. 0.2.4.6.81
Performance
.2 .4 .6 .8 1
Competence to Perform
bandwidth = .8
Examine the relationship between competence to perform
and performance in the full sample. Does performance
increase with competence?
When health workers
work in teams,
performance can be
high even if
competence is low, but
we can see evidence
that increasing the
competence of health
workers at the lower
end can improve
performance.
But at the upper end,
improving competence
does not improve
performance, even
though average
performance is low.
86

87. How to measure?
 Many tools are available to measure process quality.
– Clinical Observations, Simulated Patients, Standardized
Patients, Paper-Based Vignettes, Tablet-Based Vignettes,
Video Vignettes, Patient Chart Audit…
 Identify the key bottlenecks.
– Observing relatively rare events is difficult and costly.
– Consider simulations and vignettes.
 Know your sample size.
– Larger countries will require larger banks of vignettes or
simulations.
– These are costly to set up, but remember that rapid data
means investing in these high startup costs.
 Ken Leonard’s work in Tanzania shows that there are many
ways of increasing attention span.
87

88. The Kyrgyz Performance Based Payments (PBP) Project:
work jointly done with Aneesa Arur, Arsen Askerov, Jed
Friedman, and Asel Sargaldakova
 Kyrgyz Republic has had persistently high (for the region) maternal and
neonatal mortality rates
– Near-universal institutional deliveries (over 95%) and coverage of
primary care services
 Hypothesis is that poor quality of care is limiting improvements in MMR and
NMR
 Project aims to improve quality of care for Maternal and Neonatal Health
(MNH)
– 3 year pilot of Performance Based Payments (PBPs) focused on quality
of MNH services at district hospitals
– Quality to be assessed by peer evaluators every quarter using a
Balanced Scorecard which includes structure, clinical care and process
measures of quality (more on this later)
– PBPs will be a dimension of Diagnosis Related Group (DRG) payments
for MNH services; Hospital Directors have autonomy over use
– In addition, hospitals expected to also receive performance feedback as
part of the PBP intervention package
88

89. Measuring Quality of Maternal and Newborn
Care
 The study uses data from the baseline survey of the PBP Impact
Evaluation
 This survey was conducted in all 63 Rayon Territorial Hospitals and
Centers of General Practice in the Kyrgyz Republic.
 Instruments included:
1.Health facility assessments: Hospital assessment and ANC
checklist
2.Simulated patients for post partum hemorrhage and neonatal
asphyxia
3.Direct observations of deliveries and antenatal care visits
4.Clinical record audits for normal deliveries, complicated
deliveries, stroke, AMI, neonatal asphyxia
5.Patient exit interviews
 All components used structured (quantitative) questionnaires or
checklists to collect data, and all field workers were trained clinicians
89

90. Direct Observation: Labor and Delivery
90
92%
78%
80%
45%
40%
87%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Palpates uterus 15 minutes after delivery of
placenta
Takes mother’s vital signs 15 minutes after
birth
Tasks for second and third stage of labor [4]
Complications during previous pregnancies
[3]
Danger signs [2]
General tasks for initial client assessment [1]
[1] Checks clients card or asks client her age, length of pregnancy, and parity, Takes temperature, Takes pulse, Asks/notes
amount of urine output, Performs general examination (e.g. for anemia, edema), Performs abdominal examination: checks
fundal height with measuring tape, Performs abdominal examination: checks fetal presentation by palpation of abdomen,
Performs abdominal examination: checks fetal heart rate with fetoscope/ultrasound, Performs vaginal examination (cervical
dilation, fetal descent, position, membranes, meconium)
[2] Fever, Foul smelling discharge, Headaches or blurred vision, Swollen Face or Hands, Convulsions or loss of consciousness,
Shortness of breath, Vaginal bleeding
[3] High blood pressure, Convulsions, Heavy bleeding during or after delivery / hemorrhage, previous c-section, Prior stillbirth,
Prolonged labor, Prior neonatal death, Abortion, Prior assisted delivery
[4] Supports perineum as baby's head is delivered, Assesses completeness of the placenta and membranes, Assesses for perineal
and vaginal lacerations

91. Pre-eclampsia/eclampsia Knowledge Test
91
51%
78%
62%
74%
92%
68%
58%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Wrong: Actively Restrain
Wrong: Give Intravenous Diazepam
MeanActions To Take If Presented With
Convulsion [2]
Action to take: stabilize with Anti-Hypertensives
Action to take: stabilize with Magnesium Sulfate
Proper Diagnosis: Severe Pre-Eclampsia
Mean Examination Actions [1]
[1] Time Of Onset Of Present Symptoms, Level Of Consciousness, Any Convulsions, Check Vital Signs (Temp, Bp,
Pulse, Respirations), Listen To / Assess Fetal Heart Tones, Fetal Movement, Check Urine Protein
[2] Administer Oxygen At 4-6 L Per Minute If Available, Place In Side Lying Position, Protect From Injury, Give
Magnesium Sulfate, Provide Anti-Hypertensives (Nifedipine Or Apresoline), Actions To Take If Presented With
Convulsion: Mean

92. Comparing Patient Exit Interviews with
Direct Observations
92
0.000
0.100
0.200
0.300
0.400
0.500
0.600
0.700
0.800
0.900
1.000
HIV status Blood
pressure
Urine test Augment Episiotomy Timing of
Meds
Dry Skin-to-skin Covered
Initial Client Assessment Intermittent
Observation
of First Stage
Labor
Continuous Observation
of Second and Third Stage
Immediate Care
Exit Interview: Unobserved Exit Interview: Observed Direct Observation

93. Measuring Quality of Maternal and Newborn
Care
 Administrative data from all 63 RTHs and CGPs on
preventable maternal and neonatal complications that are
targeted by Kyrgyz RBF pilot.
 Extracted data on ICD-10 codes used for DRG payments on:
 Perineal lacerations
 Post-partum hemorrhage
 Other obstetric trauma
 Birth asphyxia
 We calculate rates of delivery and neonatal complications for
both types of hospitals and test the various measures of QoC
from the survey data against these complications rates to see
which measures are more predictive of complications rates.
93

94. Summary of Key Findings*
1.Instruments appear to be better suited to predicting complications
rates for Territorial Hospitals rather than Centers of General Practice.
2.Criterion-based Clinical Audits do not appear to be predictive of
hospital quality, particularly in Kyrgyzstan where meticulous
documentation was not incentivized prior to the RBF pilot.
3.Direct Observations perform better in terms of having the expected
sign on the correlation, but are often not significant predictors of
QoC.
4.Simulations using the MamaNatalie anatomical model were more
predictive of the administrative maternal and neonatal complications
rates.
 This finding is important from a policy perspective because
training and evaluations of provider skill as well as IEs can use
this relatively inexpensive tool.
94

95. Some Caveats
1.While we use data on the case mix treated by these hospitals, and
consider preventable complications that are targeted by the Kyrgyz
RBF pilot:
a)Our results may be driven by the fact that complicated cases are
systematically referred to some of these hospitals.
b)However, the cadre of hospitals considered here is not the type
that patients are referred to.
c)Further, we attempted to select complications that were less likely
to be screened through antenatal care.
d)In addition, we account for hospital type in the analysis.
2.Further, unobservable third factors may lead to certain areas having
less healthy populations
3.Certain complications may also be beyond the control of the hospital
and may instead be a factor of the quality of ANC.
95

96. RBF and Quality of Care:
What the impact evaluations are telling us

97. Evidence base for RBF and QoC is slim
 Das et al. (2016) systematically review the published literature and find 8
studies that explore RBF impacts on QoC with methodological rigor
 Wide variation in the studies
– Burundi, DRC, Egypt, Philippines, and Rwanda
– 3 RCTs, 4 dif-in-difs, 1 propensity matched case-control
– 5 focused on PHCs, 2 on district hospitals, 1 on both
– 3 directly incentivized limited set of quality indicators, 3 utilized
composite quality index (BSC)
– 3 directly paid health workers, 4 paid facilities
– Incentives ranged from 5% to 275% of base salary
– Measurement of quality includes hosuehold interview, patient exit
interview, record review, direct observation, and vignette responses
97

98. Evidence base for RBF and QoC (II)
 Wide variation in the findings:
– Structural quality: very mixed findings
▫ Increase in number of qualified staff and drug availability in DRC 1
▫ Increase in clinical knowledge in Philippines
▫ However majority of cases find little change
– Process quality: some gains in ANC processes
▫ History taking, blood tests, urine tests increased in Egypt
▫ Summary process quality score improves by 0.2 SDs in Rwanda
▫ However no change in DRC, and no measurement in other studies
– Quality outcomes: again, mixed findings
▫ Improved patient knowledge in Egypt and DRC
▫ Improved client satisfaction in DRC 1 and Burundi but not DRC 2
▫ Little change in assessed health outcomes (nutritional status of U5s improves
in Rwanda)
98

99. Evidence base for RBF and QoC (III)
 Very difficult to generalize from current evidence base
– Diversity of program design and involvement of QoC
– Most evaluations not primarily concerned with QoC
 Despite several programs granting autonomy and funds to
enhance structural quality, evidence of improvement is
minimal
– Procurement and managerial bottlenecks?
 Does increase in utilization negatively spillover onto QoC?
 These mixed findings call for deeper investigation into
– Design of RBF programs
– Implementation of programs
99

100. Evidence base for RBF and QoC (IV)
 RBF impact evaluation portfolio is expected to
generate much more evidence (eventually over 30
country studies)
 Let’s review in-depth results from two recently
completed studies:
– Zambia
– Zimbabwe
100

101. Both Zambia and Zimbabwe saw gains in
select targeted coverage measures
 Delivery
– In-facility deliveries increased 12.8 percentage points in Zambia
– 13.4 pp increase in Zimbabwe
 ANC and PNC
– Concomitant gains in PNC in both countries
– No gain in ANC coverage in either country
 Family planning
– No gains in Zambia
– 12 pp increase in Zimbabwe, only among women with primary
education or below
 Child health
– No improvements in vaccination coverage in Zimbabwe
– 6-7 pp increase in select vaccination measures in Zambia
– 4 pp reduction in extreme stunting in Zimbabwe
101

102. Zambia: Structural Quality
• Little change in individual measures of structural quality, however an
aggregate index suggests gains in RBF compared with pure control
districts
• Gains in structural quality of care-specific indices
102
RBF vs. Control 1 RBF vs. Control 2
Impact
estimate
p-value
Impact
estimate
p-
value
Facility experiences no power outage -0.019 0.881 0.194 0.159
Facility experiences no water outage 0.041 0.688 0.051 0.476
Infrastructure index 0.195 0.470 0.483* 0.099
RBF vs. Control 1 RBF vs. Control 2
Impact estimate p-value Impact estimate p-value
Curative Care 0.39 0.204 0.28** 0.042
Family planning 0.15 0.578 0.08 0.546
Delivery Room 0.61** 0.010 0.57*** 0.000

103. Zambia: Quality of ANC
• Process measures of ANC quality for a few measures are
improved in RBF as compared to C1 and C2, but little gain in
overall index
• Household survey results suggest 3 percentage point
increase in IPT coverage: a directly targeted process quality
indicator
103
RBF vs. Control 1 RBF vs. Control 2
Impact estimate p-value
Impact
estimate
p-
value
Weighed -0.02 0.632 0.06 0.251
Blood pressure measured -0.03 0.809 0.08 0.452
Abdomen measured 0.07 0.152 0.09* 0.063
Abdomen palpated 0.00 0.987 0.12* 0.083
Advice on diet 0.14*** 0.009 0.02 0.850
Quality of ANC index 0.02 0.921 0.33 0.165

104. Zambia: Quality of child health care
• No apparent gain in process quality of child health visit
104
RBF vs. Control 1 RBF vs. Control 2
Impact
estimate
p-value
Impact
estimate
p-value
Asked age -0.01 0.880 0.02 0.776
Weighed child -0.07 0.378 0.06 0.498
Measured height -0.10 0.104 -0.02 0.577
Physically examined -0.09 0.327 -0.08 0.350
Quality of care index -0.09 0.669 0.14 0.565

105. Zambia: Satisfaction on ANC
• Higher levels of patient satisfaction in selected dimensions
of ANC (but not all) in RBF as compared to the two
controls
• Little apparent increase in overall satisfaction
105
RBF vs. Control 1 RBF vs. Control 2
Impact
estimate
p-value
Impact
estimate
p-value
The health worker spent a sufficient amount of time
with the patient 0.08* 0.067 0.08* 0.081
You trust the health worker completely in this health
facility 0.07* 0.066 0.03 0.569
Satisfaction index 0.04 0.826 0.12 0.574

106. Zambia: Satisfaction on child health care
• Little apparent increase in overall satisfaction for
child care
106
RBF vs. Control 1 RBF vs. Control 2
Impact
estimate
p-value
Impact
estimate
p-value
The amount of time you spent waiting to be seen by
a health provider was reasonable -0.02 0.823 -0.06 0.477
You trust the health worker completely in this health
facility 0.11* 0.057 0.04 0.504
Satisfaction index 0.09 0.617 0.04 0.858

107. Zimbabwe: Structural Quality
 Improvements in select measures of structural
quality:
 Higher incidence of biomedical waste disposal (16
% points; p = 0.027)
 Increased availability of iron (16 pp), folic acid (21
pp), and urine dipsticks (42 pp)
 Increased availability of select equipment
 electric autoclave (29 pp) and refrigerator (27 pp)
 However no gains in majority of measures
107

108. Structural Quality - Mapping of Checklist
 Elements from the quality checklist were
extracted from the facility survey
instrument and assigned the same weight
to calculate the indices.
Process Quality – ANC (Household Survey)
Zimbabwe: Structural and Process Quality
108
Impact
estimate
p-value
Administration and planning 0.167 0.674
Medicines and sundries stock
management
0.017 0.969
Out Patient Department 0.468 0.213
Family and Child Health 0.837** 0.021
Maternity Service 0.009 0.981
Referral services 0.182 0.667
Community services 0.049 0.866
Infection control and waste
management
0.492 0.272
Impact estimate p-value
Blood pressure measured 0.025 0.570
Urine sample taken 0.153** 0.027
Blood sample taken 0.084 0.129
Any tetanus injection 0.075* 0.056
Number of tetanus injections 0.312* 0.063
Any iron taken 0.003 0.951
Number of days iron taken -1.161 0.868
Anti-parasite drugs taken 0.031 0.117
Malaria prophylaxis taken 0.033 0.654

109.  Quality of service indicators recorded
in the HMIS also show significant
increases
Zimbabwe: Process quality in the HMIS
109
 Even for indicators that show no
significant increase from patient
recall data.

110. Main takeaways and priority questions
 Systematic review and two country studies suggest
– RBF is effective to improve process quality of ante natal care
– Very mixed results on structural quality and client satisfaction
– Little evidence in either direction on (a) quality of other processes, (b) long
run health outcomes
 Challenges with QoC improvements suggest need to revisit how QoC is
measured and incentivized under RBF
 Scope to revisit efficiency of RBF spending: reallocate funds away from
coverage indicators where coverage is already high and towards quality
indicators
 Combine RBF with complementary investments in quality improvement (e.g.
CQI) to amplify RBF impacts on quality?
 Incentivize activities involved in the facility management of quality?
110

111. THANK YOU