This presentation was given at the pay-for-performance workshop in Tanzania, November 2015

1. Understanding causal pathways within health
systems policy evaluation through mediation
analysis: an application to payment for
performance (P4P) in Tanzania
Laura Anselmi, Peter Binyaruka, Masuma Mamdani
Josephine Borghi
Payment for Performance: a health systems perspective
A workshop for scientists and practitioners
Dar es Salaam, 26 November 2015

2. Rationale
• Programme evaluation has mainly focused on measuring the
impact on outcomes, with little attention to the causal pathways
• The relevance of undertaking process evaluation to be integrated
with outcome evaluation is increasingly recognised
• Process evaluation is particularly relevant for complex
interventions:
 It increases confidence in the plausibility of outcome effects
 It increases the external validity of the evaluation
• Process evaluations have been carried out, but without formal
assessment of causal pathways

3. Causal mediation analysis
• A causal mechanism is a process through which a programme or
intervention influences an outcome
• It can be identified by specifying intermediate outcomes or variables
(mediators) that are on the causal pathway between intervention and
outcome
• Causal mediation analysis has been employed to test change
pathways within the evaluation of public health programmes
• Mediators have been limited to individual level indicators,
psychological or physical
• Health system mediators which are relevant to the evaluation of
health systems or services interventions have not been considered

4. Program Outcome
Mediator 1
P4P Indirect
effect
P4P Measured effect =
P4P Direct Effect + P4P Indirect effect 1 + P4P Indirect Effect 2
Confounder
Mediator 2
P4P indirect
effect
Sequential ignorability b):
Given treatment status and
pre-treatment confounders
the mediators are ignorable
(no confounders affecting
both mediators and outcome)
Programme
Measured effect
Confounder
Confounder
Causal mediation analysis
Sequential ignorability a):
Given pre-treatment confounders
the treatment is assigned
independently of potential
outcomes and mediators
P4P
direct effect

5. P4P Programme in Tanzania
• P4P Scheme introduced in 2011 by the MoH in the Pwani region
• Target on Maternal and Child Health Care outcomes
• Outcome evaluation:
• 8.2% increase in coverage of institutional deliveries (ID)
• 6.5% increase in delivery in public health facilities
• 10.3% increase in the uptake of two doses of anti-malarial (IPT) during
pregnancy
• The effect of P4P on a number of governance, financing and
human resources factors has been identified
• Data:
• Health facility survey: 150 HFs (75 P4P + 75 control)
• 1-2 interviews with health workers per HF
• 1,500 exit interviews
• 3,000 household survey
• Baseline: Jan-March 2012, Endline: February 2013

6. P4P theory of change in a
Causal Mediation Analyses Framework
P4P Outcome
P4P indirect effect
through Governance
P4P indirect effect
through Human
Resources
P4P indirect effect
through Financing
P4P direct
effect

7. Methods
• Step1: Estimating the impact of P4P on outcomes (DiD)
P4Pt indicator of P4P district
δt time indicator
Xijt women socio-economic characteristics
γj HF fixed effects
• Step 2: Identifying effect of P4P on potential mediators (DiD)
• Step 3: Identifying direct and indirect causal effects (DiD)
𝛽1
3
P4P direct effect
𝛽1
2
X 𝛽4
3
P4P indirect effect through mediator M
𝑌𝑖𝑗𝑡 = 𝛽0
3
+ 𝛽1
3
(𝑃4𝑃𝑗 × 𝛿𝑡) + 𝛽2
3
𝛿𝑡 + 𝛽3
3
𝑋𝑖𝑗𝑡 + 𝛽4
3
𝑀𝑖𝑗𝑡 + 𝛾𝑗 + 𝜀𝑖𝑗𝑡
3
𝑌𝑖𝑗𝑡 = 𝛽0
1
+ 𝛽1
1
(𝑃4𝑃𝑗 × 𝛿𝑡) + 𝛽2
1
𝛿𝑡 + 𝛽3
1
𝑋𝑖𝑗𝑡 + 𝛾𝑗 + 𝜀𝑖𝑗𝑡
1
𝑀𝑖𝑗𝑡 = 𝛽0
2
+ 𝛽1
2
(𝑃4𝑃𝑗 × 𝛿𝑡) + 𝛽2
2
𝛿𝑡 + 𝛽3
2
𝑋𝑖𝑗𝑡 + 𝛾𝑗 + 𝜀𝑖𝑗𝑡
2

8. Results: Potential mediators (Step 2)
Potential mediators Effect of P4P (% change)
Financing
Proportion of women who paid for delivery in a HF (1) -8.0**
Proportion of women who paid for delivery in a public HF (1) -7.5***BS
Service delivery disrupted due to broken equipment last 90days -149**
Drug stock-out index-general (0-1 index) -17.2***BS
Medical supplies stock-out index (0-1 index) -14.8***BS
Oxytocin injection stock-out last 90days -36.2***BS
Ergometrine injection stock-out last 90days -26.1**
Drugs at delivery stock-out index (0-1 index) -27.0***BS
Mean all financing indicators(0-1 index) -8.3**
Factor analysis weighted score (2) -60.0***BS
Governance
Max time from external supervision: 90 days ago -18.0**
Dist/Regional supervision provided positive feed-back 23.8**
Dist/Regional supervision provided negative feed-back 28.2**
Dist/Regional supervision delivered supply -19.3**
Dist/Regional checked records 1.5**
Dist/Regional observed consultation 0.8**
Human resources
Mean patient satisfaction with interpersonal care (0-1 scale) (1) 6.7***BS
Mean kindness ranks for HW at delivery (0-1 scale)(1) 10.3***BS
* p<0.10, ** p<0.05, *** p<0.01 , BS: Significant at 5% level with Bonferroni adjusted p-value for multiple outcomes: Bonferroni adjusted p-value
Financing 0.0047, Governance 0.0017, Human resources 0.0414, (1) Out of all women deliveringin a HF in same catchment area (2) equipment,
vaccines, drugs, medical supply

9. Results: P4P direct and indirect effect (Step 3)
• Facility based delivery
P4P total effect: +8.2%
P4P indirect effect through mean of all financing indicators: +1 %
P4P indirect effect through reduction in stock-out of oxytocine: +1.8 %
P4P direct effect: +7.2% or +6.4%
• Delivery in public health facility
P4P total effect: +6.5 %
P4P indirect effect through reduction in stock-out of oxytocine: +1.9 %
P4P direct effect: +4.6%
• Uptake of two doses of IPT during pregnancy
P4P total effect: +10.3 %
P4P indirect effect through reduction in last supervision being 90 days ago: +1.5 %
P4P direct effect: +8.8%

10. Sensitivity analysis
• Semiparametric mediation analysis to quantify the sensitivity of results
to the assumption of no confounders affecting mediator and outcome
• Analysis carried out at the HF level
• Estimate a logit model for binary mediators
• Multiple hypothesis testing adjustment of p-values for families of
mediators
• DiD with district fixed effects

11. Summary: Indirect effects of P4P
• P4P significantly affects a number of financing, governance and
human resources factors which could potentially mediate its
effect on maternal care outcomes
• The effect of P4P on the reduction of oxytocine injection stock-
out mediates the effect of P4P on institutional deliveries (22%)
and deliveries in a public health facility (30%)
• The effect of P4P on the frequency of supervisions mediates 15%
of the effect of P4P on the uptake of at least two doses of IPT
during pregnancy

12. Some reflections on mediation analysis
• Mediation analysis rarely applied using difference in difference
• Quasi–experimental setting + Difference-in-Difference provide
confidence that the assumption of no pre-treatment
confounders is satisfied
• But how plausible are the assumptions of no confounders
between mediator and outcome?
• Data availability limits testing pre-trends for mediators
• Possible differences in results according to the level of the
analysis
• Little analysis of the role of individual level factors or other
moderating factors
• Possibly simplified description of the causal chain

13. Conclusions
• Mediation analysis is helpful to quantify causal direct and
indirect effects and the relative relevance of change
pathways
• It requires assumptions to identify causality and these can
not be tested formally
• Quantify the P4P indirect effects helps in thinking about
relative cost-effectiveness compared to alternative
interventions

14. Thank you!
Acknowledgements to the whole “P4P team”