1. 7/12/2017
1
Module 2
Illustrate conceptual models and performance frameworks for
healthcare systems
Anbrasi Edward
Christian Connections for International Health
Pre-conference Workshop July 13, 2017
Linking Mission to Objectives
Develop
Operational
Objectives
Set
Organizational
Goals
Develop
Value based
Strategic
Vision
and
Mission
Craft and
Execute
strategy of
activities
Improve/
Change
Revise as
required
Revise as
required
Improve/
Change
Corrective
action
Step 1 Step 2 Step 3 Step 4 Step 5
Monitor
with Key
Performance
indicators
2. 7/12/2017
2
Road Map
Formulating the research/program questions, Theory of
change
Configuring frameworks, determining indicators, data
sources
Study design, sampling, data collection strategies
Managing the Evaluation
Analysis
Sharing, Reporting, Use, Learning, Action Plans
Configuring Performance Frameworks
Country-demand driven and country led
Build on existing in-country processes and experience
Audience: inclusive, involve civil society and other stakeholders
Align with leading global innovations
System framework with a value chain and validated indicators
(avoid collinearity )
Address epidemiological transition
Measurement within the scope of the health organization
Maturity and sophistication
Mandate, skills, resources and independence to succeed
Measures, methods, frequency, sampling, weighting
Financing and cost structure
3. 7/12/2017
3
Considerations for Evaluation
Program context, objectives, intervention strategies
External vs participatory, quantitative and formative
Process and/or outcome measures
Data sources: clinical obs, record review, interviews
Enumeration, sample size, design effect, power calculations
Sampling: simple random, systematic, stratified, cluster (multistage),
non-probability, convenience, LQAS
Clinical, managerial, cost effectiveness measures
Other considerations
Primary sampling unit, unit of analysis
Informed consent procedures, ethical considerations
Resources: Time, Personnel, Finance
Theory of Change
http://www.mhealthknowledge.org/sites/default/files/MAMA_Global_MEPlan_FINAL_all_0.pdf
4. 7/12/2017
4
Hierarchy of Research Methods?
Robust experimental designs to weaker qualitative techniques?
Contextual application; no right and wrong approach,
appropriateness of the strategy, knowledge base, resource
availability, time, environment, level of analysis.
M&E Overview
Performance indicators
The logical framework approach
Formal surveys
Rapid appraisal methods
Participatory methods
Public expenditure tracking surveys
Cost-benefit and cost-effectiveness analysis
Impact evaluation
Not a comprehensive list! Some are complimentary
Broad and narrow applicability
Depends on the need of users; the speed for information and cost
5. 7/12/2017
5
Performance Indicators
Measures of inputs, processes, outputs, outcomes, and impacts for
development projects, programs, or strategies
Appropriate study design with formal surveys, analysis and reporting,
indicators enable managers to track progress, demonstrate results,
and take corrective action to improve service delivery
Setting performance targets and assessing progress toward
achieving them
Provides an early warning system to allow corrective action to be
taken
Effective means to measure progress toward objectives, facilitates
benchmarking comparisons between different organizational units,
districts, and over time
Tendency to include too many indicators, or those without adequate
data sources resulting in expensive, impractical, and underutilized
data
Monitoring Indicators
Selecting indicators, determining baselines, setting targets - fundamental
aspects of monitoring
Choosing appropriate number of indicators, well-defined and accurate
monitoring can increase data quality used for reporting and decision-making
Data informs learning agenda, provides evaluation teams with necessary
information to understand what project or activity results have been achieved
Include priorities and existing efforts of governments, implementing partners,
and other donors, to the extent possible, to align efforts, reduce data
collection and reporting burdens
Monitoring plans should be reviewed on a regular basis to ensure that
selected indicators continue to be relevant and useful for management
needs – evolving
Disaggregated data improve understanding of the progress toward
achievements that an indicator captures, by providing details of the
experiences of subsets of beneficiaries
6. 7/12/2017
6
Monitoring Indicators
To ensure the provision of consistent, timely, and high quality data,
each indicator is required to have an authentic reference
Includes indicator definition, source of data, frequency of data
collection, and disaggregation etc
If data are not of good quality, can be misleading and result in wrong
decisions
Consider; where will the data come from, what level of quality are the
data expected to be, how will data be gathered and stored to protect
integrity, and the privacy of the participants?
Direct or 3rd party sources (Government ministry, other)?
Data Quality, Storage Security
Validity; Data clearly and adequately represent the intended result.
Integrity; Data have safeguards to minimize the risk of transcription
error or data manipulation.
Precision; Data have sufficient level of detail to permit management
decision making.
Reliability; Data reflect consistent collection processes and analysis
methods over time.
Timeliness; Data are available at a useful frequency, are current,
and timely enough to influence management decision making
Proper data storage and security are critical to protecting data
integrity, optimizing data usability, and safeguarding potentially
sensitive or personally identifiable information.
7. 7/12/2017
7
Global Reference List
guide monitoring of health results nationally and globally
reduce excessive and duplicative reporting requirements
enhance efficiency of data collection investments in countries
enhance availability and quality of data on results
improve transparency and accountability
Disaggregations that include equity stratifiers (e.g. age and sex, geography,
socioeconomic status, place of residence)
Data sources: civil registration, vital statistics systems, population-based
health surveys, facility-generated data, routine facility information systems,
health facility assessments and surveys, administrative data sources such as
financial and human resources information systems
Results Chain – Refer to HO
8. 7/12/2017
8
Log Frames
Helps to clarify objectives of any project, program, or policy
Aids in the identification of the expected causal links—the “program
logic” results chain: inputs, processes, outputs (including coverage or
“reach” across beneficiary groups), outcomes, and impact
Leads to the identification of performance indicators, risks which
might impede the attainment of the objectives
Used to engage stakeholders in clarifying and SMARTening
objectives and designing activities
Management tool to review progress and take corrective action
Can impede creativity and needs periodic updating to accommodate
HC environment changes
Example from PEPFAR
Formal Surveys
used to collect standardized information from a sample of individuals
or households
Provides baseline data against which the performance of the
strategy, program, or project can be compared
Comparing different groups at a given point in time, changes over
time in the same group, actual conditions with the targets established
in a program or project design
Describing conditions in a particular community
Providing a key input to a formal evaluation of the impact of a
program or project
Findings from population sample can be applied to the wider target
group or the population as a whole
9. 7/12/2017
9
Household Surveys
Method
Unicef MICS Multi Indicator Cluster Surveys
USAID, DHS Demographic Health Survey
KPC 30 Cluster Surveys Knowledge Practice Coverage
PEPFAR, PMI President’s Initiatives for AIDS, Malaria
WHO Verbal Autopsy Verbal Autopsy for Non Clinicians
CS-KPC Rapid Catch Child Survival Key Performance Indicators
PETS, WB Public Expenditure tracking surveys
LSMS Living Standards Measurement Surveys
Indonesia FLS Indonesia Family Life Surveys
HEIDE Household expenditure and income data for transitional economies
IHSN International household survey network
DDP WB Development data platform
Facility Performance Measures
Method Description
Service Provision Assessment
DHS Measure
National Sample, 3-5y
Service Availability Mapping, WHO Mapping of service and resource availability, National
and district level decision making, Annual
Health Facility Census, JICA Health assets mapping, National, 5y
FASQ, Measure Facility Audit of Service Quality
QIQ, Measure Quick Investigation of Quality
LQAS, (EPI, HIV/AIDS) Lot Quality Assurance Sampling
IMCI- MCE, WHO, Rapid HFA Focus on child health
GFATM, RBM, EPI, RBF etc Global Initiatives
Modules include system readiness, QoC, provider and client interviews, donor assistance, social
marketing, record reviews, costing etc
Technical Areas: MNCH, STI, TB, HIV/AIDS, FP etc
Management, financing, motivation, design, sampling, duration and types of data collection methods vary
10. 7/12/2017
10
Rapid Service Quality Checklist
• Health Facility ____________
• Observer ____________
• Supervisor _____________
• Date _____________
• Did the service provider
1. Use a sterile needle for each injection? Yes No
2. Use a sterile syringe for each injection? Yes No
3. Give the child all the required vaccinations? Yes No
4. Record the vaccination on the child health card? Yes No
• Outreach sessions
• Were vaccines transported in cold boxes with ice packs? Yes No
Rapid Appraisal
Rapid low-cost ways to obtain feedback of participants and other
stakeholders
Provides real time information for management decision-making,
especially at the project or program level
Providing qualitative understanding of complex socioeconomic
changes, highly interactive social situations, or people’s values,
motivations, and reactions
Providing context and interpretation for quantitative data collected by
more formal methods
Findings usually relate to specific communities or localities—thus
difficult to generalize from findings
11. 7/12/2017
11
Rapid Appraisal Methods
Key Informant Interviews: a series of open-ended questions posed to
individuals selected for their knowledge and experience in a topic of
interest. Interviews are qualitative, in-depth, and semi-structured and
use guides with a list topics or questions
Focus Group Discussions; a facilitated discussion among 8–12
carefully selected participants with similar backgrounds. Participants
might be in the study, program staff, or stakeholders. The facilitator
uses a discussion guide, note-takers record comments and
observations
Direct Observation; use of a detailed observation form to record what
is seen and heard at a program site.
Mini-survey; structured questionnaire with a limited number of close-
ended questions that is administered to 50–75 people. Selection of
respondents may be random or ‘purposive’
Participatory Evaluation
Collaborative problem solving
Contributes to learning and leads to strengthened program design/
implementation.
External expert plays a facilitating role with stakeholders
Share ideas and encourage team to consider options.
12. 7/12/2017
12
Participatory Methods
Learning about local conditions and local people’s perspectives
and priorities to design more responsive and sustainable
interventions
Identifying problems and trouble-shooting problems during
implementation
Evaluating a project, program, or policy – Mixed methods studies
Providing knowledge and skills to empower poor people
Stakeholder analysis; develop an understanding of the power
relation ships, influence, and interests of the various people
involved in an activity and to determine who should participate,
and when
Mixed Methods Evaluation
In addition to determining impact, it is equally important to understand the
implementation process of the program, their successes and challenges to
bring about improvements in quality of the program and learning for program
improvement and expansion
Therefore, it is critical that the impact evaluation and formative research are
systematically aligned
13. 7/12/2017
13
Cost-Benefit and Cost-Effectiveness Analysis
Cost-benefit analysis measures both inputs and outputs in monetary
terms. Cost-effectiveness analysis estimates inputs in monetary
terms and outcomes in non-monetary quantitative terms
Informing decisions about the most efficient allocation of resources
Identifying projects that offer the highest rate of return on investment
Impact Evaluation
Systematic identification of the effects; positive/negative, intended/ not, on
individual households, institutions, and the environment caused by a
program or project
Helps determine equity, effectiveness, coverage, and outcomes
Range from large scale sample surveys and control groups are compared
before and after, to small-scale rapid assessment
Measuring outcomes and impacts of an activity and distinguishing these
from the influence of other, external factors.
Accountability of investments, informing decisions on whether to expand,
modify or eliminate projects, programs or policies
Drawing lessons for improving the design and management of future
activities
Comparing the effectiveness of alternative interventions
Expensive, time consuming
14. 7/12/2017
14
Study design Definition
Randomized control
trial (RCT)
Involves random allocation of the intervention and comparison (e.g. usual care) to different study groups, including
measurement of the outcome before and after the intervention has been made.
Non-RCT
Observations are made before and after the implementation of an intervention, both in a group that receives the
intervention and in a control group that does not.
Case-Control
Compares people with a specific outcome of interest (‘cases’) with people from the same source population but
without that outcome (‘controls’), to examine the association between the outcome and prior exposure (e.g. having
an intervention).
Cohort
Participants who do not have the outcome of interest are allocated (but not randomly) to receive alternative
treatment/therapy/interventions (experimental group) or the standard of care or conventional treatment
(comparison group) or no intervention (control group) and then followed up to determine whether they experience
the outcome of interest.
Cohort
One group (pre- and post-): Single selected group under observation, with a careful measurement being done
before applying the experimental treatment and then measuring after. Two groups (pre- and post-): Group of
people with a common set of characteristics or a set of characteristics that are followed up for a period of time to
determine the incidence of an outcome.
Interrupted time-
series
Uses observations at multiple time points before and after an intervention (the ‘interruption’) in an effort to detect
whether the intervention has had an effect significantly greater than any underlying trend over time.
Hierarchy of Study Designs
Study design Definition
Case or time series
Observations are made on a series of individuals, usually all receiving the same intervention, before and after an
intervention but with no control group.
Cross-sectional
Both exposures and outcomes are measured at a single point in time, and the prevalence of the outcome is compared
among those with and without exposure.
Quantitative, Other
Systematic empirical investigation of social phenomena via statistical, mathematical or numerical data or
computational techniques other than those listed above.
Cost-effectiveness
Cost Effectiveness Analysis (CEA) is a type of economic evaluation that examines both the costs and health
outcomes of alternative intervention strategies.
Mixed Methods
Combines data collection approaches, sometimes both qualitative and quantitative, into the study methodology. Some
studies combine study designs, whereas others may have a single overarching research design, but use mixed
methods for data collection.
Implementation Science
Scientific study of methods to promote the systematic uptake of research findings and other evidence-based practices
into routine practice, and to improve the quality (effectiveness, reliability, safety, appropriateness, equity, efficiency) of
care. It includes the study of factors impacting behavior at the individual (i.e. provider, patient) and organizational
levels.
Case Study
The study of selected current phenomenon over time to provide an in-depth description of essential dimensions and
processes of the phenomenon. Case studies may involve both qualitative and quantitative data and may focus on
single or multiple cases.
Participatory action
research (PAR)
Systematic inquiry, with the collaboration of those affected by the issue being studied, for purposes of education and
taking action or effecting change.
Qualitative
Used to generate an understanding of complex, unquantifiable phenomena, such as people’s experiences or
perceptions.
15. 7/12/2017
15
Models of Impact Evaluation – Refer to HO
Sample Size and Power determinations
An important consideration for any evaluation study is sample size
Power analysis should be done in order to estimate the sample size needed
to detect program effects with a reasonable level of certainty
The power analysis can take account of various kinds of outcome indicators
(such as continuous, binary, and count measures) and the consequent
statistical requirements for rejecting the null hypothesis assuming small,
moderate or large effect sizes, as appropriate
Standard power calculations can be done using widely available software
The longitudinal nature of the measures proposed may require large
samples to accommodate more independent and dependent variables, and
to account for anticipated attrition from the study cohort
In Impact evaluations timing is essential. Many outcomes of interest may
take months to demonstrate change; Maternal Mortality, Illness Incidence etc
16. 7/12/2017
16
Baseline Data
Collect data on demographic and socioeconomic characteristics of
women/households
Basic information on health outcomes based on stage of pregnancy,
newborn care or child health
If opting to use an RCT design, then it is important that the baseline
information is collected prior to randomizing women into either the treatment
or control group
The purpose of the baseline data collection is twofold
First, to assess whether or not the women in the treatment and control group are
comparable
Second, it will provide insight into their health, behavioral patterns, and other
demographic information prior to enrolling
RCT, minimize selection bias
Data Analysis
Data may require some processing and analysis to ensure they are accurate
and make sense, but many data require substantial analysis to reach a state
where they are usable and ready to be incorporated into a report
The kind of analysis necessary depends on the kinds of data that were
collected and how those data are intended to be used
Qualitative data will undergo content or pattern analyses to see trends
Quantitative data may be simple analyses to generate sums or averages,
disaggregated, or they require complex approaches such as regression
analyses or multilevel modeling (logistic, Poisson etc)
Uses dependent variables designed to reflect the appropriate level of
measurement for each outcome of interest; binary events, such as birth in a
clinic, or other events, such as number of antenatal clinic visits and number
of birth complications, may be measured with count variables
Include data analysis plan, software etc
17. 7/12/2017
17
Presentation and Use of Data
Data will be presented using a variety of tools including tables, graphs, and
charts as appropriate
Key findings will be summarized in PowerPoint presentations
The data can presented at the monthly and quarterly meetings within the
project, and annually with partners
The primary use of the data will be to inform the program management if
changes are required for program activities
Evaluation Components
Health System and District team
Capacity building: training- technical/managerial/HIS
Utilization patterns, Quality of Care, equity, efficiency
Supply chain management, facility preparedness
Community linkages
Cost recovery
Scope, cost, data propriety
PVO
Capacity building; training, field visits etc
Communication, reports
Staff competency, performance, satisfaction; oversight, personnel
development, compensation systems etc
Financial management
Organizational learning and leadership
18. 7/12/2017
18
Evaluation Components
Program Effectiveness
Achievement of goals and objectives
Quantitative: MICS/KPC
Qualitative: KII, FGD etc
Community capacity/Health System capacity
sustainability strategy, scale up, gender,country ownership,
‘smart’ integration etc
Community
Participation, empowerment, equity
Functional community structures: CHW, VHC, Health Facility
Councils and other interest groups etc
Performance Assessment Levels
Household Surveys
Health Facility Assessments
Hospital Accreditation and Performance Audits
Health System Performance
19. 7/12/2017
19
Health System Performance
Assessment Models
WHO Health Systems Monitoring
The World Health Organization (WHO). Health System Metrics. Report of a Technical Meeting. Glion, Switzerland, 28-29 September 2006.
20. 7/12/2017
20
WHO Health Systems Building Blocks
WHO. Everybody's business. Strengthening health systems to improve health outcomes. WHO's Framework for Action. Geneva. 2007
IHP+
21. 7/12/2017
21
USAID’s Health System Assessment: A
How to Manual 2007
Health Indicator Framework: Canada
Canadian Institute for Health Information, Health Indicators, 2005.
22. 7/12/2017
22
Australia’s National Health Performance Framework
National Health Performance Committee (NHPC) (2001), National Health Performance Framework Report, Queensland Health, Brisbane.
Duran, A. - Assessing Performance of the
National Health System in Spain
23. 7/12/2017
23
Frenk J Dimensions of Health System
Reform, Mexico
Health Policy 27 (1994): 19-34
Guisett, A. Portugal
Functions
the system
performs
Health System
Stewardship
Service Delivery
Health
Financing
Resource
Generation
Ultimate
Goals of the
system
Improved
Health (Level
and Equity)
Responsiveness
Social and
Financial Risk
Protection
Improved
Efficiency
Intermediary
objectives of
the system
Improving
Access
Improving
Coverage
Delivering High
Quality, Safe
Health Services
Healthy behaviours
& Health Promotion
Tackling
Broader Health
Determinants
External Context
Demographic Legal and regulatoryEconomic
Epidemiological Technological Socio‐cultural
Political
Source: Adapted from WHR 2000; Atun 2005
24. 7/12/2017
24
Atun et al. - Multimethods Evaluation
Primary Health Care Reforms in Estonia
Atun R, Menabde N, Saluvere K, Jesse M, Habicht J. “Introducing a Complex Health Innovation-Primary Health Care Reforms in Estonia
(multimethods evaluation)”. Health Policy 79 (2006): 79-91.
WHO’s Performance Assessment Tool for Quality
Improvement in Hospitals (PATH)
Veillard J. et al. A performance assessment framework for hospitals: the WHO regional office for Europe PATH project. Int J Qual Health Care. 2005;
17(6):487-496
25. 7/12/2017
25
EFQM - European Foundation for Quality Management -
Fundamental Concepts of Excellence to Health Care
Based on the PATH Framework
World Health Organization (WHO). 2003. Measuring Hospital Performance to Improve the Quality of Care in Europe: A Need for
Clarifying the Concepts and Defining the Main Dimensions. Copenhagen: WHO Regional office for Europe.
Baldrige Award
The Baldrige National Quality Program. Criteria for Performance Excellence 2009-2010. National Institute of Standards and Technology, Gaithersburg,
MD.
26. 7/12/2017
26
Handler et al, Conceptual Framework to Measure
Performance of the Public Health System
Handler, Issel and Turnhock. “A Conceptual Framework to Measure Performance of the Public Health System”. AJPH 2001; 8: 1235-1239.
Kruk et al, Assessing Health System Performance in
Developing Countries
Kruk ME, Freedman LP. Assessing health system performance in developing countries: A review of the literature. Health Policy 85
(2008) 263-276.
27. 7/12/2017
27
International Development Research Center
Organizational Assessment A Framework for Improving Performance Lusthaus C, Adrien M, Anderson G, Carden F, and Montalván GP
Value Based Healthcare - Sisters of Saint Mary Hospital
1st Recipients of the Malcolm Baldridge National Quality Award
SSM Health Care’s Vision Statement:
Through our participation in the healing ministry of
Jesus Christ, communities, especially those that are
economically, physically and socially marginalized,
will experience improved health in mind, body, spirit
and environment within the financial limits of the
system
SSM Health Care’s Mission Statement:
Through our exceptional health care services, we
reveal the healing presence of God.
SSM Health Care’s Values:
In accordance with the philosophy of the Franciscan
Sisters of Mary, we value the sacredness and dignity
of each person. Therefore, we find these five values
consistent with both our heritage and ministerial
priorities:
. Compassion
. Respect
. Excellence
. Stewardship
. Community
28. 7/12/2017
28
Balanced Scorecard
– “A multi-dimensional framework for describing,
implementing and managing strategy at all levels of an
enterprise by linking objectives, initiatives, and measures to
an organization’s strategy.”
Kaplan & Norton, 1996
29. 7/12/2017
29
Distinctive Advantage of ‘Balanced’ Scorecard?
Complex health sector with varied stakeholders
Pts, community, civil society, providers, HCO, insurers,
financers, policy makers, politicians
Distortions created by global agendas for vertical programs
Increasing demand for accountability
“Strategic performance drivers”
Efficient management systems
Generating Scorecards
Groene et al” “The balanced scorecard of acute settings : development process, definition of 20 strategic objectives and
implementation. ISQuA, Aug 2009
33. 7/12/2017
33
Accelerated Reconstruction Efforts to Address Disease Burden
Myriad of challenges impeding
service delivery
Ranks among the highest for
maternal and child mortality
Total public health spending
escalated to S$280 million (‘08-’09),
85% financed by external donor
assistance
Service delivery primarily through
contracting mechanisms
Critical shortage of skilled health
providers estimated at 39% (BPHS)
Demand for female providers who
currently constitute <24% of the
workforce
BSC Design - BPHS and EPHS
Maternal and Newborn Health
Child Health and Immunization
Public Nutrition
Communicable Diseases
Mental Health
Disability
Essential Medicines
Blood Transfusion Services to
Support BPHS
Medicine
Pediatrics
Surgery
Obstetrics and Gynecology
34. 7/12/2017
34
NHSPA Study Design
Stratified random sampling of upto
25 HF in each province
3 District Hospitals
7 Comprehensive Health Centers
15 Basic Health Centers
Case management observations
and exit interviews on a systematic
random sample of 5 patients <5y and
5 patients ≥5y
Interviews with 5 Providers
randomly selected in each facility
Capacity Assessments:
infrastructure, services, equipment,
drugs, guidelines, etc
Study Sample in Provinces Assessed btw 2004
and 2008
Sample 2004 2005 2006 2007 2008
Provinces 33 30 30 30 29
Facilities
551 612 612 615 600
U5 Patient Observations 2787 2827 2912 2982 2962
>5y Patient Observations 2630 2947 2965 3008 2921
Exit Interviews 5053 5716 5857 5895 5796
Provider Interviews 1388 1418 1666 1827 2175
1 Daikundi province was not included in 2004 as it lacked functional facilities, but was included in the later years.
Helmand, Kandahar, Zabul and Uruzgan were excluded after 2004 and Farah was also excluded in 2008.
35. 7/12/2017
35
69
Stakeholder (MOPH, Donor, NGO) Consensus on:
Purpose of Balanced Scorecard
Measurement Domains
Unit of analysis
Process & frequency of review/decisions
Principles for benchmarking
Short-listing indicators based on validity, importance,
reliability
Designing the Scorecard
Afghanistan BSC
Measure
National
Median
Lower
Benchmark
Upper
Benchmark Badakhshan Badghis Baghlan Balkh Bamyan Farah
A. Patients & Community
1 Overall Patient Satisfaction % 83.1 66.4 90.9 86.4 76.9 90.9 84.7 97.9 82.8
2 Patient Perception of Quality Index % 76.0 66.2 83.9 77.6 66.2 82.2 80.0 84.4 69.4
3 Written Shura-e-sehie activities in community % 34.2 18.1 66.5 35.6 0.0 34.2 17.7 34.5 73.2
B. Staff
4 Health Worker Satisfaction Index % 63.5 56.1 67.9 63.5 57.6 67.9 68.3 61.4 54.4
5 Salary payments current % 76.7 52.4 92.0 54.9 91.8 45.8 53.3 91.4 97.7
C. Capacity for Service Provision
6 Equipment Functionality Index* % 65.7 61.3 90.0 69.6 62.2 57.5 67.3 75.8 66.3
7 Drug Availability Index % 71.1 53.3 81.8 52.9 50.1 72.8 56.1 85.6 9.8
8 Family Planning Availability Index % 61.4 43.4 80.3 54.2 57.9 70.4 64.9 82.7 0.0
9 Laboratory Functionality Index (Hospitals & CHCs) % 18.3 5.6 31.7 31.7 3.8 15.2 0.0 37.0 0.0
10 Staffing Index -- Meeting minimum staff guidelines % 39.3 10.1 54.0 38.0 22.4 42.7 45.8 53.0 57.1
11 Provider Knowledge Score % 53.5 44.8 62.3 48.6 41.6 49.3 54.0 69.0 45.5
12 Staff received training in last year % 39.0 30.1 56.3 68.9 50.9 39.0 52.4 35.5 37.2
13 HMIS Use Index % 67.7 49.6 80.7 60.9 62.7 40.0 72.9 67.7 72.4
14 Clinical Guidelines Index % 34.8 22.5 51.0 18.3 25.5 29.9 16.4 41.9 59.5
15 Infrastructure Index % 55.0 49.3 63.2 63.2 49.7 50.0 58.3 57.9 76.7
16 Patient Record Index % 65.6 56.1 92.5 51.5 98.5 80.7 97.3 64.5 97.1
17 Facilities having TB register % 15.8 8.3 26.6 32.5 27.0 16.1 16.4 0.0 4.3
D. Service Provision
18 Patient History and Physical Exam Index % 70.6 55.1 83.5 54.2 71.7 55.1 85.4 83.6 52.0
19 Patient Counseling Index % 29.6 23.3 48.9 23.3 40.4 29.3 55.3 33.2 16.0
20 Proper sharps disposal % 62.2 34.1 85.0 64.4 34.1 76.9 75.1 85.0 67.8
21 Average new outpatient visits per month (BHC>750 visits) % 22.2 6.7 57.1 27.3 10.0 27.3 71.4 22.2 0.0
22 Time spent with patient (> 9 minutes) % 18.0 3.5 31.2 21.0 30.7 1.2 27.3 12.8 18.0
23 BPHS facilities providing antenatal care % 62.0 28.9 82.8 28.9 49.4 49.7 67.2 88.1 82.8
24 Delivery care according to BPHS % 25.4 10.5 39.3 38.0 36.2 10.5 39.3 38.0 57.1
E. Financial Systems
25 Facilities with user fee guidelines % 90.6 80.3 100.0 94.8 95.6 95.9 28.9 86.1 100.0
26 Facilities with exemptions for poor patients % 84.7 64.4 100.0 68.5 54.6 69.3 84.3 95.6 93.9
F. Overall Vision
27 Females as % of new outpatients % 55.2 46.5 59.7 46.9 45.9 56.0 55.1 55.2 59.0
28 Outpatient visit concentration index CI (-1 to 1) -0.010 0.041 -0.055 0.021 0.024 -0.038 0.025 -0.076 -0.036
29 Patient satisfaction concentration index CI (-1 to 1) 0.002 0.020 -0.018 -0.019 0.000 0.003 -0.007 -0.005 0.020
Composite Scores
30 Upper Benchmarks Achieved % 17.2 10.3 30.8 17.2 6.9 6.9 20.7 34.5 31.0
31 Lower Benchmarks Achieved % 82.8 75.9 89.7 86.2 79.3 86.2 86.2 96.6 72.4
36. 7/12/2017
36
Performance by BSC Domains: National Median Score
↑19*** ↑13.7*
↑11*
↑27***
↑29***
* p<0.05 *** p<0.0001
Edward A, Kumar B, Kakar F, Salehi AS, Burnham G, et al. (2011) Configuring Balanced Scorecards for Measuring Health System Performance:
Evidence from 5 Years’ Evaluation in Afghanistan. PLoS Med 8(7): e1001066. doi:10.1371/journal.pmed.1001066
Domain A: Patient and Community
37. 7/12/2017
37
Domain B: Provider Satisfaction
Domain C: Capacity for Service Delivery
0
20
40
60
80
100
2004 2005 2006 2007 2008
Eqipment Functionality
Drug Index
FP Index
Lab Functionality
Infrastructure Index
clinical guidelines
38. 7/12/2017
38
Domain C: Capacity for Service Delivery cont..
Domain D: Service Provision
0
20
40
60
80
100
2004 2005 2006 2007 2008
Pt Screening
Pt Counseling
Sharps disposal
Ave new outpt
visit/m (BHC 750)
Consultation time
(≥ 10m)
ANC services
Delivery Services
39. 7/12/2017
39
Hospital BSC Trends - 2007 to 2009/10
11.1*
5.8
4.8 3
8.6*
6*
7.6*
* p<0.05
Cascading the Scorecard Lower Upper National Median
2004 2005 2006 2007 2008
A. Patients & Community
1 Overall Patient Satisfaction
66.4 90.9 83.1 86.3 86.0 77.7 81.0
2 Patient Perception of Quality Index
66.2 83.9 76.0 76.2 80.3 77.6 77.5
3
Written Shura-e-sehie activities in
community
18.1 66.5 34.2 54.5 66.4 86.0 94.3
B. Staff
4 Health Worker Satisfaction Index
56.1 67.9 63.5 64.1 68.1 69.0 69.1
5 Salary payments current
52.4 92.0 76.7 90.0 81.3 90.7 82.7
C. Capacity for Service Provision
6 Equipment Functionality Index
61.3 90.0 65.7 67.0 78.7 83.8 88.4
7 Drug Availability Index
53.3 81.8 71.1 83.7 85.7 81.0 86.3
8 Family Planning Availability Index
43.4 80.3 61.4 70.0 82.9 93.7 94.9
9
Laboratory Functionality Index
(Hospitals & CHCs)
5.6 31.7 18.3 36.3 43.3 58.5 64.5
1
0
Staffing Index -- Meeting minimum
staff guidelines
10.1 54.0 39.3 58.0 66.9 63.9 72.1
1
1 Provider Knowledge Score
44.8 62.3 53.5 69.0 68.7 68.7
1
2 Staff received training in last year
30.1 56.3 39.0 74.3 68.9 68.5 71.1
1
3 HMIS Index
49.6 80.7 67.7 65.8 74.9 91.5 92.4
1
4 Clinical Guidelines Index
22.5 51.0 34.8 48.9 61.6 78.3 83.9
1
5 Infrastructure Index
49.3 63.2 55.0 44.6 48.7 54.6 54.1
1
6 Patient Record Index
56.1 92.5 65.6 63.2 69.4 70.0 69.9
1
7 Facilities having TB register
8.3 26.6 15.8 20.6 37.4 53.7 62.9
D. Service Provision
National
SC
NGO/Provincial SC
Facility/Team SC
Two-Way Influence
Two-Way Influence
Two-Way Influence
Facility and QI Team SC
Select Indicators
Provincial SC for managers and NGOs
at the provincial level to benchmark
and improve performance
Organizational SC with all domains
and indices for policy and planning at
the national level
Community SC will include
indicators of CHW/Shura
activities, referral and
feedback on quality of
services
Community SC
BSC
CSC
40. 7/12/2017
40
Preparatory Groundwork
Community Gathering
Performance Scorecard Input Tracking Scorecard
•Divide into focus groups
• Information on entitlements/budgets
• Develop input indicators
• Collect evidence on input use
•Record data
•Divide into focus groups
•Develop performance indicators
•Finalize indicators (5‐8 max.)
•Performance scoring by groups
•Verify High/Low Scores
•Record data
Self‐Evaluation
Scorecard
Empowerment
Transparency
Development
Accountability
Improved health care
and utilization
Feedback and Dialogue
Interface Meeting
Trust
Efficiency
Community Scorecards
Edward, A., K. Osei-Bonsu, et al. (2015). "Enhancing governance and health system accountability for people centered
healthcare: an exploratory study of community scorecards in Afghanistan." BMC Health Serv Res 15: 299.
Siadara Facility - CSC
Indicators Round 1 Round 2
(3m)
Round 3
(6m)
Reasons
Provider SC
Water (dysfunctional water pump) 5 10 10 Water pump is repaired
Lack and non-use of IEC materials 8 10 9.5 Additional IEC materials and old copies replaced with new
Damaged clinic roof due to snow
avalanche
9 8 10 Metal roof of clinic was repaired
Inadequate medical equipment 7 6 7 The medical equipment requires replacement, very old
Clinic cleanliness 7 9 9.5 Right now even the clinic guard is aware of preparing 0.5 % chlorine
solution. But incinerator is not available
Waiting time 9 10 10 patient load is low, separate OPD for male and females
Clinic management 9 9.5 10 We work as a team, interpersonal relationship is good, clinic is
managed responsibly
Accurate clinical examination 7 9 9.5 When patient volume is high, the examination time is not satisfactory
especially for ANC, PNC and IMCI
Good behavior among the
personnel
10 10 10 No problem, we work as a team
Community SC
No Water in the clinic 5 10 10 Water pump is repaired and safe water is available for all
Electricity in clinic 3 6 10 Initially no electricity, the Provincial Rehabilitation Team (PRT)
promised to install solar power and eventually did install it.
Medicines 7 10 10 Quality is good, but inadequate quantity. They prescribe ‘white tablets’
(Paracetamol) for all conditions. Now clinic has all medicines required
for each patient for the condition
Clinic staff adequacy 10 10 10 Adequate staff and always present on the job
Waiting room seating 5 10 10 Patients stand due to insufficient chairs. More chairs were provided.
Staff Punctuality 10 10 10 They are always present
Staff behavior with patients 10 10 10 Good behavior of staff, all are satisfied with them
Patient Counseling 10 10 10 When we get treatment, they give good counseling on meds, return
Patients Waiting Time 10 10 10 We do not wait for long time we are satisfied
Clinic cleanliness 10 10 10 Clinic is clean, no flies, mosquitos, and they disposed the garbage
41. 7/12/2017
41
Siadara Action Plan
Indicator Action Proposed Who? Timeline Implementation Research Observations
Water supply Request sent to AADA by Shura following
the monthly meeting with the facility staff.
The clinic in charge coordinated it through
the Yakawlang district governor, Bamyan
PPHD and NGO
Clinic in
charge, AADA,
PPHD, Shura
3m after the
shura
meeting
Multiple negotiation meetings and follow up.
Staff paid for water pump from salary and
eventually reimbursed
IEC materials Clinic in charge requests staff to prepare a
list of IEC materials and forward the request
to NGO
Clinic in
charge, NGO
1m Follow up requests for 2m. but materials
were not received.
Condition of clinic
Building (damaged
roof)
Clinic in charge address the issue with
NGO and PPHD
Clinic in
charge, NGO,
PPHD
1-6m The DHO and District Governor were also
engaged in processing the request
Clinic equipment
and infrastructure
(waiting room chair)
Clinic in charge requests all departments
for equipment needs, and submits requests
to NGO
Clinic in
charge, NGO
3m Discovered chairs in storage and
transferred to waiting area. Though shura
were not indicated in action plan there were
involved in all decisions.
Clinic hygiene Clinic in charge to train staff on infection
prevention (IP), create a plan for the clinic,
and follow up
Clinic in
charge and
staff
Ongoing URC NGO working on QA, set standards for
IP, trained the staff and helped with action
plan for different sections of the clinic. The
staff followed up daily. The staff engaged in
a self-audit, till they achieved 100%
compliance, and were awaiting an external
evaluation by URC
Accuracy of exam ,
pt education and
time spent with pt
Pt triaging to avoid ‘noise’ , time spent with
pt must be considered
staff Ongoing According t BPHS, spend at least 9min with
pts The guard was involved in determining
the reason for clinic visit and guides the pts
to the specific area
BSC in Zambia
Better health Outcome through Mentoring and Assessment (BHOMA)
project in 3 Rural Districts1
3 Strategy Approach
District Strategy: QI Teams
HF Strategy: Establish clinical care standards, self assessment
reports, leadership training
Community Strategy: 60 CHWs ($60/m), Health Committees,
TBAs
Linked Facility Surveys: HF and HHS
7 Domains; Pt and Community, HR, Service capacity, Finance
(action plan), Governance, Service Provision + Overall Vision
Mutale et al; Measuring Health System Strengthening: Application of the BSC approach to rank baseline performance of three rural districts in
Zambia; PLOS; March 2013; Vol 1
42. 7/12/2017
42
PAHO Renewing PHC in the Americas
Multiphase Strategy for Design of a 3-Tier BSC
Phase Phase
1
•JHU: Selected Review of HSPA Models, June 2010
Phase Phase
2
•PAHO HSPA Experts Meeting , June 8‐9: 2010
Phase Phase
3
•JHU/PAHO TAG Review of Expert Recommendations, Prioritize HSPA domains and indicators, July 27‐29, 2010
Phase Phase
4
•JHU Design of PAHO HSPA Conceptual Model and BSC Tool
Phase Phase
5
•JHU and PAHO TAG Review and Development of HSPA Model and 51 Indicator score Card
Phase Phase
6
•PAHO Country Stakeholder Meeting ‐ Cuba, December, 2010
Phase Phase
7
•JHU/PAHO TAG Review of Cuba Meeting Recommendations and Proposed Changes to BSC
Phase Phase
8
•JHU Redesign of Conceptual Model and 118 Indicator BSC
Phase Phase
9
•JHU and PAHO TAG Meeting: Review of Revised HSPA and BSC Levels for Country Implementation, December 14,
2011 – 3‐Tier 110 Multi‐indicator framework
Phase Phase
10
•Country Consultation and Contextualization using Delphi tool and Pilot of PAHO BSC, April 2013
I ‘feared’ when they mentioned
a BSC for HSP
Minister of Health,
El Salvador
43. 7/12/2017
43
Executing the BSC
5 Principles
Translating Strategy into operational terms
Align the organization to the strategy
Make strategy everyone's everyday job
Make strategy a continual process and
Mobilize leadership for change
3 Functions of the BSC
Measurement system
Strategic Management system
Communication tool
PAHO PHC Oriented BSC
45. 7/12/2017
45
World Vision Research Study
5 year Multi-country Multi Site
Mixed Methods Research Study
on Impact of Integrated Package
of Community Based Interventions
for MNCH
2012-2017, $3.3M in partnership
with 4 Country Research
Institutions
Sustainable Development Goals & CHNIS
46. 7/12/2017
46
Disease Burden for IMCI Conditions, Immunization, Nutrition
Outcomes
Reported (past 2 weeks) Cambodia
(n=3813) %
Guatemala
(n=4096) %
Kenya
(n=6407) %
Zambia
(n=2648) %
Diarrhea 23.3 17.8 8.0 16.1
Cough or difficult breathing 19.9 15.7 5.4 9.7
Fever 46.8 22.1 15.1 22.2
Full Immunization 56.1 66.5 39.3 63.1
Measles Vaccination 62.9 70.8 54.6 70.8
Underweight (<-2SD) 39.9 26.9 14.8 11.1
Severe Underweight (<-3SD) 19.6 11.5 5.8 2.6
Stunted (<-2SD) 47.5 48.3 40.3 37.4
Severe Stunting (<-3SD) 28.6 20.7 22.8 14.7
Wasted (<-2SD) 6.0 10.6 11.3 3.8
Severe Wasting (<-3SD) 1.4 5.1 5.8 1.3
Anemic (<11g/dl) 74.8 - 57.2 -
Severe Anemia (<7g/dl) 1.5 - 4.0 -
Quality of Screening and Assessment, IMCI
Guatemala Cambodia Kenya Zambia
Variables N % N % N % n %
Clinical Observations of Sick Children 35 39 164 100
Primary complaints
Cough 16 45.7 7 17.9 44 27.3 23 23.0
Difficulty breathing 2 5.7 2 5.1 5 3.1 5 5.0
Diarrhea 4 11.4 3 7.7 19 11.8 39 39.0
Fever 8 22.9 20 51.3 127 78.9 31 31.0
Vomiting 1 2.9 na na 14 8.7 1 1.0
Provider Asked/Checked
Age of child 35 100 39 100 157 95.7 88 88.9
Duration of complaint 34 100 38 97.4 161 98.2 97 97.0
Child weighed 25 71.4 31 79.5 145 88.4 43 43.0
Height measured 21 60.0 1 2.6 28 17.1 16 16.2
Check weight against growth chart 19 55.9 6 15.4 36 22.0 28 28.3
Assessed nutritional status with MUAC 11 32.4 2 5.1 19 11.6 4 4.0
Sought care from CHWs 1 2.8 6 15.3 3 1.8 12 12
Mean consultation time (minutes) 30 12.6 37 10.5 161 8.5 98 8.2
47. 7/12/2017
47
Child Caretaker Counseling
WV Zambia Impact Analysis ScorecardComparison Intervention Difference of Difference
Baseline Final Baseline Final
DOD
Mean
SE p-valueMean/% SE Mean/% SE Mean/% SE Mean/% SE
A. Demographics
Mean family size 4.7 2.0 5.0 1.9 5.3 2.4 5.5 2.5 -0.2 0.1 0.163
Mean assets 2.4 1.7 2.7 1.5 2.3 1.8 2.3 1.6 -0.3 0.1 0.002
Head of HH ever attended school 90.4 0.92 87.2 1.0 93.1 0.8 93.7 0.8 3.9 1.8 0.029
Health insurance2 - - 1.3 0.1 - - 0.6 0.1 0.0 - -
Household severe hunger 1.6 0.4 7.6 2.0 7.2 0.7 18.7 1.9 5.6 2.4 0.020
Access to adequate sanitation1,2,3 80.6 1.1 98.3 0.4 82.3 1.1 94.0 0.7 -6.0 1.7 0.000
Access to clean water1,2,3 10.8 0.9 99.1 0.3 38.6 1.4 93.5 0.7 -33.4 1.8 0.000
World Vision child sponsorship 17.3 0.9 7.6 0.8 5.9 0.6 3.7 0.6 7.5 1.5 0.000
B.1. Child Health and Nutrition
Prevalence of ARI in past 2 weeks 4.6 0.6 4.1 0.6 14.8 1.0 4.9 0.7 -9.4 1.5 0.000
Prevalence of fever in past 2 weeks 11.2 0.9 19.4 1.2 33.2 1.3 16.6 1.2 -24.9 2.3 0.000
Prevalence of diarrhea in past 2 weeks 14.5 1.0 16.8 1.1 17.7 1.0 13.0 1.0 -7.1 2.1 0.001
Care seeking for ARI 1,3 25.0 12.5 33.3 7.9 33.7 4.6 84.6 5.8 42.6 17.4 0.015
Care seeking for fever 26.5 6.3 58.5 3.9 25.7 2.8 86.2 3.2 28.5 8.7 0.001
Care seeking for diarrhea 73.5 3.2 41.5 3.6 65.4 3.1 51.1 4.3 17.7 7.2 0.013
Child use of ITN in the previous night1,3 54.0 1.4 78.2 1.3 55.9 1.4 42.7 1.5 -37.5 2.8 0.000
Exclusive breastfeeding <6 months3 100.0 0.00 66.2 3.8 100.0 0.0 67.9 3.6 1.6 5.7 0.778
Breastmilk and complementary feeding 6-9 months 6.1 2.9 69.4 5.9 4.7 2.6 53.4 5.9 -14.6 9.4 0.122
Antibiotic treatment of ARI3 62.3 6.2 74.4 6.7 41.1 3.5 28.0 6.4 -25.2 12.2 0.040
Antimalarial treatment of fever 32.5 4.2 11.5 2.2 25.9 2.5 24.9 3.3 20.0 6.1 0.001
Children with diarrhea given ORT1,3 100.0 0.0 82.2 3.0 100.0 0.0 86.5 3.0 4.3 3.6 0.231
Underweight prevalence1 10.1 0.8 10.8 1.0 12.0 0.8 11.7 1.0 -1.0 1.8 0.595
Stunted prevalence1,2, 3 39.1 1.3 34.5 1.6 35.9 1.2 34.7 1.6 3.3 2.9 0.247
Wasted prevalence1,2, 3 3.6 0.5 4.5 0.6 3.9 0.5 3.5 0.6 -1.4 1.1 0.224
Severe acute malnutrition 1.4 0.3 1.6 0.4 1.3 0.3 1.7 0.4 0.1 0.7 0.851
Full vaccination3a 69.4 2.8 69.6 3.1 56.7 3.1 56.3 3.0 -0.6 6.0 0.920
Measles vaccination1 91.4 1.7 99.5 0.5 87.1 2.1 98.0 0.9 2.8 3.1 0.231
Received first dose vitamin A b 69.8 2.8 97.0 1.1 67.3 2.9 88.5 2.0 -6.0 4.8 0.210
DPT vaccination (3 doses) 1,3 78.0 2.5 91.5 1.9 68.1 2.9 90.9 1.9 9.3 4.9 0.056
B.2. Maternal and Neonatal Health
Antenatal care (ANC) index3 3.3 0.9 3.3 1.0 3.0 1.0 3.9 0.7 0.8 0.1 0.000
Newborn care index3 2.7 0.5 2.8 0.5 2.7 0.5 2.9 0.3 0.1 0.0 0.117
Contraceptive prevalence rate1,3 62.7 1.7 70.5 2.7 59.3 1.9 59.9 2.7 -7.2 4.7 0.124
Mean month of first ANC visit 4.8 8.0 3.8 1.0 4.7 5.9 4.2 1.2 0.5 0.5 0.257
At least 1 ANC visit1,3 95.5 0.8 78.4 1.8 98.7 0.5 97.4 0.6 15.8 2.0 0.000
4 or more ANC visits1, 3 48.4 2.0 76.0 2.1 53.8 2.1 60.6 2.0 -20.8 4.2 0.000
Skilled provider for ANC 94.7 0.9 97.6 0.7 96.7 0.8 97.1 0.7 -2.6 1.5 0.091
Facility based birth 21.4 1.7 65.3 2.0 24.7 1.9 83.4 1.5 14.8 3.5 0.000
Skilled attendant at birth1,2, 3 82.8 1.6 95.5 0.9 79.0 1.8 94.3 0.9 2.5 2.6 0.338
PNC less than 1 day after delivery1,3 10.4 1.4 15.1 1.7 7.3 1.2 58.3 2.0 46.3 3.5 0.000
Skilled provider for PNC 98.5 0.6 99.1 0.4 97.5 0.8 99.0 0.4 0.9 1.1 0.414
C. Quality of Care
Children U5 mean utilization (previous month) 231.2 162.4 198.2 266.9 239.6 138.3 259.4 246.0 52.8 136.4 0.701
Mean supervision visits in last 6 months4 3.3 2.3 7.0 5.4 2.2 1.5 11.0 9.8 5.1 3.8 0.186
Service provision score3 6.4 0.7 5.2 0.4 6.4 0.5 5.3 0.7 0.2 0.4 0.681
Essential medicines score3,4 5.8 0.4 6.0 0.0 5.9 0.3 6.0 0.0 -0.1 0.2 0.447
Emergency obstetric care services score1,3 3.4 1.6 4.9 0.3 4.3 0.9 5.0 0.0 -0.7 0.6 0.231
Children U5 Patient assessment score 3.6 1.5 4.4 2.5 6.0 1.6 3.6 2.2 -3.3 0.6 0.000
Children U5 Patient counseling score 1.8 1.4 4.5 1.2 3.2 1.7 4.2 1.8 -1.7 0.4 0.000
Children U5 Caretaker satisfaction score4 6.6 2.0 6.5 1.0 6.3 1.4 7.0 1.1 0.9 0.4 0.037
48. 7/12/2017
48
Performance Metrics
Lord Kelvin
“You cannot manage what you cannot measure”
Albert Einstein
“Not everything that can be counted counts and not
everything that counts can be counted”
Future considerations…
49. 7/12/2017
49
Church as an integral part of the
healthcare system
ONS Happiness Indices
Overall, how satisfied are you with your life nowadays?
Overall, to what extent do you feel the things you do in
your life are worthwhile?
Overall, how happy did you feel yesterday?
Overall, how anxious did you feel yesterday?
All answered using a 0 to 10 scale where 0 is 'not at all'
and 10 is 'completely'.
50. 7/12/2017
50
Re‐imagining the 21st Century Physician: IHI
Health care is locked in a 100-year-old paradigm. We gained over
30 years of life expectancy in the 20th century, but only five of
those years are directly attributable to medical care. So how can
we break out of this costly, reactive, biomedical, and disease-
oriented approach ---- and focus instead on the key factors that
contribute to good health? One answer is to change the way
doctors are trained
IHI Open School
What does the new physician look like?
What new competencies do medical professionals need?
And what disruptive innovations can help our current health care
system adapt to a new paradigm?