A presentation from the 2014 Annual Results and Impact Evaluation Workshop for RBF, held in Buenos Aires, Argentina.

1. Strengthening Quality of Heath
Services in RBF Projects
Dinesh Nair, Ronald Mutasa,
Rianna MohammedPatron Mafaune,
Kathleen Hill

2. Presentation Outline
 Session objectives and overview
 Integrating Quality into RBF projects
 Quality in global health
 Principles/approaches to improving quality
 Integrating quality into RBF projects
 Prioritizing health conditions
 Selecting standards and defining quality measures
 Measuring and verifying quality
 Aligning RBF with QI & Health System Strengthening efforts
 Country Examples: Liberia & Zimbabwe

3. Session Objectives
 Understand importance of quality and gain
familiarity with improvement approaches
 Understand common operational needs and
strategies of RBF projects that incentivize quality
 Understand experience to date in RBF projects that
incentivize quality

4. 4
The Issue of Quality in Health Care
High quality care is……
Effective: Adherent with evidence-based
standards
Safe: does not harm patients
Client centered: Respectful of patient
needs, values & preferences
Equitable: Does not vary in quality
because of personal characteristics
(gender, ethnicity, SES, etc)
IOM, 2001, Crossingthe Quality Chasm

5. Preliminary Results from Service Delivery Indicator
(SDI) Survey: Nigeria
Sample of 1,172 health facilities in 6 states (one from each
geo-political zone) end 2013 – not yet public
 Diagnostic accuracy less than 40% (judged by vignettes) -
less than half that observed in Kenya
 Adherence to guidelines< 1/3rd (by vignettes)
 Knowledge of maternal & neonatal complications<
20%, <40% of what it is in Kenya

6. Closing the Know-Do-Gap Between Proven Best Practices and Actual
Practice in the Places Patients Receive Care
6

7. Conceptualizing Quality
(Source: Donabedian)
7

8. Principles Underlying Improvement
 Effective teamwork
 Understanding how processes of care function within a system and critical
bottlenecks
 Use of data to continuously measure
and track progress
 Understanding and
focus on patient needs
 Regular shared learning
for rapid dissemination
best practices 8

9. Prioritizing What to Improve
Deciding How to Measure if Care Is Improving
9
 Choosing improvement
Aims: “What are we trying to
accomplish?”
 Defining quality measures:
“How will we know if a change
is an improvement?”

10. 10
Improvement Team Reviewing Quality Indicators

11. Measuring Quality to Improve Care:
Who Needs What Data for What Purpose?
Facility Staff (managers, providers, staff -hospitals and clinics):
-Need quality measures to assess and continuously improve services.
Is care improving?
Regional/District & Program Managers:
-Need measures to assess and continuously strengthen essential system functions
(e.g. competent workforce).
Are essential system functions performing to standard?
--------
 Clients (users of care)
 National Policy-makers (value, policy)
 Global Stakeholders (leadership, advocacy, accountability)

12. Integrating Quality into RBF Projects: Prioritizing
Health Conditions/Services for Improvement
Focus on:
 High-burden conditions in local context (leading causes of
mortality and morbidity)
 High-burden conditions for which there is strong evidence
of effective health care interventions (preventive and
curative)
 Country government priorities: involve local decision
makers and experts
 Consider phasing improvement priorities: “impossible to
improve everything at once”

13. Involve local and international experts to:
 Review country standards against global evidence: evidence is constantly changing
 Distill standards into minimum “interventionbundles”: focus attention on essential high-
impact interventions
 Illustrative quality of care process measures based on minimum standards:
 % cases adherent with standards – “all or nothing adherence” (e.g. % PPH cases
managed per minimum standard; % cases pediatric pneumonia treated per
standard)
 Average % adherencewith minimum standards (e.g. average % adherence
with newborn sepsis case-management standards; N=30 cases)
Integrating Quality into RBF Projects: Selecting Standards and
Defining Quality of Care Measures

14. Ilustrative quality measure: Quality of Partogram
Completion (not so simple!)
Quality Measure Operational Definition
 % partograms in last quarter
completed per standard
 NUMERATOR: Number partograms
documenting cervical dilation, maternal BP,
pulse, temperature at admission and at least
every 4 hours until delivery
 DENOMINATOR: Total number of
partograms reviewed

15. Defining Quality of Care Measures (continued)
 Review quality measures being used at country and
global level; adapt or develop new measures
 Consider including measures of care coordination and
performance of essential system functions (e.g.
referral/counter-referral; supportive supervision)
 Streamline and harmonize indicators: the fewer the
better!

16. Improving adherence with post-partum care best practices: Average
compliance with PNC standards
Herat Province Afghanistan- Nine Health Centers; 2009-2011
16

17. Measuring Quality: The challenges…..
JAMA Nov 13. 2013
“Quality measurement is in rapid
flux….despite the challenges of a
rapidly expanding number of
quality measures, much of health
care remains poorly measured or
unmeasured.”

18. Measurement Methods
 Observation
 Patient interviews & questionnaires (e.g. exit interview; household survey)
 Death (and near-miss) audits
 Simulations (provider competence)
 Provider knowledge/problem-solving (e.g. vignettes/case studies)
 Routine information systems (e.g. facility chart & register audits; routine
health information system)
-Regular measurement is a core principle of all improvement, but measurement
alone will not improve care! 18

19. Common Measurement Challenges
for QI and RBF efforts
 Routine HMIS in low-resource settings include few (if any) quality/content indicators
 Primary data sources (medical records/registers) often don’t include necessary data
for constructing quality measures
 Quality of clinical procedures (e.g. newborn resuscitation) cannot be assessed from a
medical record; creative measurement approachesneeded
 Few routine indicators of performance of essential system functions (e.g. %
maternities in district with functional neonatal bag & mask at bedside)
 Weak staff data management capacity (providers and managers)

20. Routine Versus Complications Care
 Easier to measure routine best practices relevant for every
patient: often simple intervention for which a “box” can be
checked in a standard record (e.g. ENC, AMTSL; immunization)
 More difficult to measure quality of complications care
 Timely accurate diagnosis
 Stabilization and successful timely referral (primary facility)
 Prompt and ongoing treatment/monitoring (hospital)
 Discharge planning and follow up
20

21. 0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
J08 F08 M08 A08 M08 J08 J08 A08 S08 O08 N08 D08
Percentage of pre-eclampsia and eclampsia case
management standards achieved
Jan-Dec 2008, average of 120 cases analyzed per month
Based on detection in 120,000 patient contacts at 31 MOH facilities
Measuring Adherence with
Pre-eclampsia/eclampsia Best Practices to Improve
PE/E Care in Niger 31 facilities
21
Changes implemented:
•Systematic screening at every
contact
•Organization daily tasks to prevent
stock outs
•Standardized emergency case
management eclampsia
•Standardized referral protocols

22. Pre-eclampsia/Eclampsia chart audit tool for
primary facility
22
Charts
Evaluation
1 2 3 4 5
1. Blood pressure (BP) recorded
1. Gestational age (GA) recorded (per one of criteria indicatedin
GUIDE)
1. Urine protein quantified (dipstick +, ++, +++)
1. Danger signs assessed (see chart review guide)
Diagnosis pre-eclampsia or eclampsia recorded if
criteria met
1. DBP > 90 and at least 2+ proteinuria pre-eclampsia(+ seizure if
eclampsia)
First Treatment and referral if primary facility
1. 4 gm loading dose of MgSO4 IV ; monitor for toxicity (reflexes,
urine output, respirations)
1. If GA < 34 weeks administer antenatal corticosteroids
1. Transfer with IV access (and provider if possible)

23. 23
Despite challenges it is possible to track quality measures from
(modified) local records (e.g. add columns to registers; stamps to
records/partograms, etc.)

24. WHO MNCH Quality of Care Consultation Geneva
December 2013: Objectives
Share global and regional experiences in the assessment
and improvement of facility MNCH QoC
Review/agree on a core and supplementary set of global
indicators for accountability - monitoring and reporting on
facility MNCH QoC
Review assessment tools, methodologies and processes
used in measuring facility MNCH QoC

25. WHO Consultation Draft Set Global Newborn Indicators:
Mix of Structural, Quality of Care Process, Outcome indicators
Proportionof health facilities with maternityservices that have functional bag& masks (2
neonatal mask sizes) in the deliveryareas
Proportionof newborns whoreceived all four elements of essential newborn care:
 immediate and thorough drying
 immediate skin-to-skincontact
 delayed cord clamping
 initiation of breastfeeding in the first hour
Proportion health facilities where Kangaroo Mother Care is operational, by level of facility
Facility neonatal mortality rate disaggregated by birth weight: >4000 g, 2500-3999 g, 2000-
2499 g, 1500-1999 g, < 1500 g
Proportionof health facilities offeringmaternityservices that have BFHI certification and
recertification not older than two years

26. Aligning RBF with QI and HSS Efforts: Challenges
and Opportunities
 RBF projects can be integrated with and/or incentivize QI and HSS efforts
 National QI strategy implementation
 Pre and in-service training & performance-based supervision
 Supply chain interventions
 HMIS improvement interventions, etc.
 RBF projects can incentivize QI activities and quality performance measures
 Regular QI team meetings
 District/facility routine collection & analysis of quality measures with action plans
 District/facility continuous quality improvement (CQI): setting aims; developing and tracking quality
measures; testing changes, sharing learning….
 Accreditation, etc.

27. Country Case Example:
Zimbabwe
 Dr. Patron Mafouna,
Provincial Medical Director,
Maniacaland Province
 Mr. Ronald Mutasa,
World Bank

28. • Focus on register completion (“# columns checked”) rather than
processes of clinical care (specific clinical interventions)
• Heavy emphasis on “structural quality” (e.g. appearance of the
facility, medications, supplies, staffing)
• Poorly defined clinical quality indicators (e.g. numerator not
clearly defined), inter-rater reliability issues
• Limited analysis and monitoring of individual checklist items
• Limited link between incentives and clinical quality of care
Zimbabwe Experience With Original Checklist Quality
Items (18 districts; Hospitals and Clinics)

29. Improving Quality Measurement Over Time:
Illustrative example “Partogram completion” indicator
RBF Project
Phase
Indicator
Checklist Guidelines for calculating
indicator
Initial Phase
% partograms
correctlycompleted
in last quarter
No criteria specified weak quality
measure
Current
Phase
% partograms
correctlycompleted
in last quarter
Randomly review 10 partograms from last
quarter Calculate % partograms documenting
specific criteria: FHR, cervical dilation, BP, pulse,
temperature documented at admission and at
least every 4 hoursuntil delivery

30. Beyond Structural Care: Phasing quality incentives for high-burden
MNCH Conditions by project phase
RBF Phase One (9-12 months) RBF Phase Two
 Routine MNH/FP
 PPH
 Obstructed labor
 Maternal & Newborn Sepsis
 Pediatric pneumonia
 Pediatric diarrhea
 Pediatric malaria
All Phase One priority conditions
plus:*
 Pre-eclampsia/Eclampsia
 Newborn Asphyxia
 Miscarriage/abortion c
management
 Management acute
malnutrition

31. Zimbabwe: Beyond structural incentives
 Innovations to introduce continuous quality improvement and
rigorously evaluate
 P4Q alone versus
 P4Q plus continuous quality improvement
 Supporting MOHCC to draft and implement
national/regional QI strategy…..align RBF with strategy
 Supporting MOHCC to introduce quality indicators into
HMIS…..strengthening HMIS

32. Zimbabwe: Lessons being learned
 Technical capacity & substance matters in P4Q, so does political
economy & building ownership by local stakeholders
 Complex trade-off between structural and clinical care indicators
 Health providers respond to P4Q indicators and incentives
 Source documents at provider level matter
 Focus on QI principles and emerging global best practices

33. Country Case Study: Liberia
 Rianna Mohammed,
World Bank

34. • High infection rates and post-surgery
complications
• No systematic use of clinical guidelines
• Accreditation scores on quality are worse
than primary facilities
Low Quality of care at Hospitals:
3
4
Hospitals characterized by:
• Poor infrastructure; inadequate
supply of drugs and equipment.
• Insufficient numbers productive,
responsive, competent staff
• Long waiting times
• Limited internal and external
financing (e.g. 85% of health
expenditure in 2009/10 directed
almost entirely at the PHC-level).
LIBERIA: Neglect of Hospitals...poor quality of care

35. ALIGNING PBF AND SYSTEM STRENGTHENING: LIBERIA PROJECT SEEKS TO
IMPROVE QUALITY VIA PBF AND COMPETENCY BUILDING
35
Hospital PBF Competency Building
Incentives for:
a) improved quality of care
(i.e. adherence to clinical
protocols)
b) quantity of services
delivered (including in-
service training to health
workers)
• Coaching and verification
support
Better competency to
improve processes of
care
• Support to development of
innovative Graduate
Medical Residency
Program.
• In-service training of
nurses, midwives and PAs
Synergistic
linkage of
components
Improved staff motivation

36. QUALITY ASSESSMENT/ MONITORING TOOLS
36
1
Complicated and assisted delivery
(including C-section)
Any labor that is made more difficult or complex by a deviation from the normal
procedure. Complicated delivery is defined as: assisted vaginal deliveries (vacuum
extraction or forceps), C-section, episiotomy and other procedures.
17
2 Normal deliveries of at risk referrals
High-risk pregnant women referred by health center to the hospital but delivered
normally. A high-risk pregnancy is defined as: evidence of edema, mal presentation,
increased BP, multi-parity, etc.
17
3
Counter referral slips returned to health
facilities
Hospital returns counter referrals letter with feedback on the referred patient to the
referring health center. The counter referral letter is completed in triplicate, with one
also given to the patient, and one retained by the hospital.
2.5
4
Newborn referred for emergency
neonatal care treatment and treated
Newborns referred for emergency neonatal care due to: perinatal complications, low
birth weight, congenital malformation, asphyxia, etc.
5
6
Referred infants and under-fives with
fever
Any surgical procedure that does not involve anesthesia or respiratory assistance. 2.5
7 Minor surgical intervention
Any surgery in which the patient must be put under general spinal/anesthesia and
given respiratory assistance. Major surgery in the case of this package of services is
defined as any of the following: Herniarraphy, Appendectomy, Myomectomy,
Sleenectomy, Salpingectomy, Hysterectomy, Thyrodectomy, Mastectomy.
5
8
Major surgery (excluding CS, including
major trauma)
Patients transferred from a lower-level facility (health center or health clinic) to the
hospital for emergency treatment.
18
9 Patients transported by ambulance 2.5
10
Number of training sessions held by
faculty for nurses, midwifes and PA
according to in-service curriculum and
defined protocols.
These indicators will incentivize the in-service training activities. 50
11
Number of nurses, midwifes and PAs
that received specialized in-service
training, relevant to benchmarks
10
Verified
Total
EarningsDefinition
Six Hospitals Total
Fee (USD)Indicators Claimed
(c) Quantity Checklist
Actual % Earned Points
1. Obstructed Labor 0.80 3.87 100% 33% 1.29
2. Hemorrhage 1.00 4.84 100% 71% 3.45
3. Maternal Sepsis 1.00 4.84 100% 50% 2.42
4. Eclampsia 0.70 3.39 100% 47% 1.59
5. Neonatal Asphyxia 1.00 4.84 100% 67% 3.23
6. Neonatal Sepsis 1.00 4.84 100% 54% 2.61
7. Prematurity 0.50 2.42 100% 47% 1.14
8. Maternal Newborn Best Practices 1.00 4.84 100% 54% 2.61
9. ETAT 1.00 4.84 100% 33% 1.61
10. Malaria 1.00 4.84 100% 71% 3.45
11. Pneumonia 1.00 4.84 100% 50% 2.42
12. Acute Diarrhea 0.80 3.87 100% 47% 1.82
13. Severe Acute Malnutrition 0.60 2.90 100% 67% 1.94
14. Surgical Safety 1.00 4.84 100% 54% 2.61
100% 60.00 100% 53% 32.20Total/Average
Childbirth:
Maternal-Newborn
Pediatric
(in-patient care)
Surgical Care
Quarter I
III. Process of Care
Detailed Score
Checklists
Weight (by
importance)
Point
Allocation
Max %
(b) Process of Care
Quality Checklists
Score
1.GENERAL MANAGEMENT (30pt)
2. HUMAN RESOURCES FOR HEALTH (16pt)
3. HYGIENE AND MEDICAL WASTE DISPOSAL (27pt)
4. DRUGS MANAGEMENT (30 pt)
5. EQUIPMENT AND SUPPLIES (84pt)
TOTAL %
Date of Verfication
TOTAL (187pt)
REPUBLIC OF LIBERIA
Ministry of Health and Social Welfare (MOHSW)
Hospital Quarterly Quality Assessment
Name of the Hospital
Name of Team Leader of Quality Verification
Verification Period
Quarterly Quality Verification Score
I. Management
II. Structural
(a) Management and
Structural Checklist
Indicators
Max Points
Actual Points
Quarter I
1. General Management 30 2.6
2. Human Resources for Health 16 9
3. Hygiene and Medical Waste Disposal 27 0
4. Drugs Management 30 8
5. Equipment and Supplies 84 48
6. Aggregated Process of Care Score 60 32
Total 247 100
Total Percentage 100% 40%
Total Quality Bonuses (USD) 159,678 64,517
PBF Bonus
Calculation Tool
Business/Operation Plan
Health Worker Bonus
Allocation
LHSSP Indices Tool for Bonus Allocation to Individual Health Workers for Hospitals
1 200 50 30 300,000 0 6,944
2 200 70 30 420,000 0 9,722
3 150 80 30 360,000 0 8,333
4 - - -
5 - - -
6 - - -
7 - - -
8 - - -
9 - - -
10 - - -
11 - - -
12 - - -
Quarter:
Total PBF Incentives Earned
% for Individual Bonus
Attendance
points [C]
Hospital Name
Total Individual Bonus
Redemption Hospital
July-Sept 2013
No Name of staff
Staff
category
Monthly
salary [A]
Perfor-
mance
points [B]
$50,000
50%
$25,000
Total points =
[A] x [B] x [C]
Indices of
the period
PBF
individual
bonus
Signature of receipt
Min
50%
Max
50%
~60
%
~20%
~20%
(1)Continuousmonitoring
(d) Impact Evaluation

37. Quality of Care Checklists for High-burden MNCH Conditions (complications):
Based on National and WHO Standards
Process of Care Checklists
Childbirth:
Maternal-Newborn
1. Obstructed Labor
2. Hemorrhage
3. Maternal Sepsis
4. Eclampsia
5. Neonatal Asphyxia
6. Neonatal Sepsis
7. Prematurity
Pediatric
(in-patient care)
8. Maternal Newborn Best Practices
9. ETAT
10. Malaria
11. Pneumonia
12. Acute Diarrhea
13. Severe Acute Malnutrition
Surgical Care 14. Surgical Safety 37

38. 38
Chart review elements (see chart review guide for specific criteria) ; each element if
recorded = 1 point
Charts
1. Admission 1 2 3 4 5
1. Cervical dilation recorded at admission (# of cm)
2. Contraction frequency and duration charted at admission
3. Fetal presentation charted at admission
4. Partograph started when cervical dilation 4 cm or greater
Admission Score (x/4)
2. Labor Monitoring (partograph)
1. Cervical dilation recorded at least every 4 hours
2. Frequency and duration contractions recorded at least every 30 minutes
3. Fetal HR recorded at least every 30 minutes
Labor Monitoring Score (x/3)
Standards for Management Obstructed Labor: Illustrative Checklist
Distilling Essential care Items (admission, labor)
Each item has chart review guide that
defines criteria
Five patient charts reviewed:
average score (% adherence best
practices) links with bonus

39. VERIFYING QUALITY MEASURES: PATIENT CHARTS REVIEWED BY AN
INDEPENDENT COUNCIL
39
Step 1: Find relevant patients
from register (e.g. “Malaria”
patient for Malaria checklist)
Step 2: Record names and
patient numbers
Step 3: Request health workers
to bring charts
• Team of minimum 2 verifiers
from Liberia Medical and
Dental Council (LMDC)
compare charts and
guides/checklists
• 5 charts each of 14 checklists
reviewed quarterly
• Total scores calculated
as % adherence for
each checklist
• Aggregated points are
tied to the performance
bonus

40. INCENTIVIZING QI ACTIONS AS PART OF HOSPITAL PERFORMANCE:
MANAGEMENT CHECKLIST
40
1.1 Performance managementstructure operational
• Hospital Board meets every month and problem-solve issues
• Senior Management Team (SMT) meets every month and problem-solve issues
• Quality Improvement (QI) or other relevant Team meets at least every month and problem-
solve issues
1.2 Business (operational) plan updated and implemented
1.3 Performance review
1.4 Reporting and filing of the key data
1.5 Financial Management
1.6 Community Involvement
1.7 Grievance mechanism
8pt
3pt
6pt
2pt
4pt
4pt
3pt
Assessment Components
Max
points

41. INTEGRATING QI ACTIONS AND RBF: QI TEAM AND COACH MOTIVATES HEALTH WORKERS TO
IMPROVE QUALITY THROUGH TRACKING QUALITY MEASURES AND COACHING
41
• Weekly/daily chart review
• Monthly management and
structural checklist review
1 Self-Assessment 2
15 20
30
40
0
50
W1 W2 W3 W4
%
e.g., Sepsis
• Post scores on a wall
• Provide detailedfeedback to
staff
Tracking and Feedback
3
• Support improvement activities (e.g., standardize chart,
waste disposal, cleaning)
• Training on the treatment protocol and chart writing
Support to Improvement QI Activities

42. Multiple levels of performancemanagement at hospital and MOHSW
levels: Incentivizing QI Actions
42
Hospital
QI team
• Responsible for improving specific quality indicators
• Carry out weekly/monthly self-quality assessments and track results at
each ward
• Provide feedback, coach and train health workers
Senior
management
Team
• Developbusinessplan with supervisors
• Hold supervisors accountable for quality indicators
• (At least ) monthly performance review with QI team & support of
QI team action plans
Hospital
Board
• Provide oversight to hospital management
• Make SMT accountablefor results and implementation
of business plan
• Community involvement and grievance
Technical
Committee • Overall oversight of performance trendsand
project implementation

43. Key lessons learned so far
• Measuring quality can be really complicated – finding a right
focus and starting from small is critical (e.g., pre-pilot, five
hospitals only, inpatient only).
• Management strengthening is essential but not easy to do – further
experiments and hands-on coaching will be important.
• Training and re-training on quality measurement is essential in
ensuring that all stakeholders understand how quality will be
measured, and trust the tools and the verifiers.
• Understanding the political economy, and being open and flexible
to change is important in building ownership among stakeholders.
43

44. THANK YOU