Presented by Beth Sutherland, David Boone and Cristina de la Torre at the MEASURE Evaluation End-of-Phase-III Event.

1. Making the Connection
Monitoring and Evaluation
in the Context of Integrated
Health Services and Systems
MEASURE Evaluation
End-of-Phase-III Event
May 22, 2014

2. Introductions
 Elizabeth Sutherland, MS, PhD
 Cristina de la Torre, MPH, ScD
 David Boone, MPH, PhD

3. Outline
 Background
 OverviewofMEASURE Evaluation’s work
inintegration
 Discussion ofMEASURE Evaluation’s workin
 Monitoring referrals
 Strengthening referralnetworks
 Integrating health informationsystems
 Take home messages and discussion

4. The Way We Were…
 What is integration?

5. The Way We Were…
 Why integration?
 What should be integrated?
 Where and how does integration happen?
 What is the value added of integration?

6. Clients Have Multiple Health Needs
 HIV
 Malaria
 TB
 Growth
 Nutrition
 Family planning
 Immunization
 Respiratory illness
 Diarrhea
 Fever

7. Making the Link Between
Clients and Services
 One stop shop
 Referrals and Referral Networks
 Community vs facility models
 Combinations

8. Making the System Work
 System made of many elements
 Together system elements support
each other
 System moves people, resources, and
information up and down the levels of the
system

9. So Where Are We Now?
 We understand that integration operates
among and within all levels of the health
system
 We know that goals and mechanisms for
integration will vary by context

10. MEASURE Evaluation’s Work
 Development and application of
standardized tools and approaches to
M&E of Integration
 Support of development of interagency
USG resources on M&E of integration
 Developments of tools and techniques for
monitoring referrals and strengthening
referral networks

11. MEASURE Evaluation’s Work
 Development and application of
standardizedtools and approaches to M&E
of Integration
 Case study approaches to documenting best
practices and lessons learned in integration
 Integrating health information systems and
using integrated data to facilitate data use

12. Referrals to Strengthen
Service Integration

13. Models of Integration

14. M&E of Referral Systems
Organizational Network Analysis Referral System Monitoring

15. Organizational Network Analysis
(ONA)
 Who is in the network
 Service gaps or redundancies
 How organizations are linked
 Information sharing
 Resource sharing
 Referrals across organizations

16. ONA Application

17. Referral Assessment and Monitoring
(RSAM) Toolkit
Guidelines for
 Establishing a routine monitoring system
of referrals
 Assessing overall functioning of the
referral system
 Can be adapted to any type of referral system

18. Focus on processes and systems
Consists of interviews and document review to
determine:
RSAM TOOLKIT
Referral System Assessment
 How the referral system is structured
 Whether referral protocols and guidelines exist
 The processes providers follow to refer and
counter-refer clients
 How well referrals are tracked and followed up
 Barriers to referral initiation and referral completion

19. RSAM TOOLKIT
Referral System Monitoring
Consists of routine data collection at facility
 How often referrals are made to different services
(initiation)
 What types of services are clients most often
referred to
 Are clients able to take advantage of the referrals
(completion)
 Is adequate follow-up provided after the fact
(counter-referral)

20. Routine Monitoring of Referral
Systems
Key indicators:
 Referral initiation
o % clients referredfromservice Ato service B
 Referral completion
o %ofreferredclients who complete referral
 Counter-referral
o % ofclients who complete referralwho areseenagain
by initiating provider

21. Referral Systems

22. COLUMN
Y
TOTAL
NUMBER
CLIENTS SEEN
AT
REFERRING
SERVICE
CLIENTS REFERRED
TO
RECEIVING SERVICES
Service 1
(FP)
Service 2
(VCT)
Service 3
(STI)
Service 4
(ART)
Service 5 Service 6
REFERRING
SERVICE
Service 1
(FP)
Service 2
(VCT)
Service 3
(STI)
Service 4
(ART)
Service 5
Service 6
(TO BE COMPLETED BY REFERRING SERVICE)
PAGE 1 of 3
Name of organization and facility: _____________________________________
Geographic unit: _______________________________
Reporting period—Month: ______ Year: __________ Prepared by: ________________________
1. Number of clients referred by type of service
Group for which data are reported—Age range: _______________ Sex: ______________

23. Illustrative Monitoring Data

24. Illustrative Data

25. Benefits of Monitoring and
Assessing Referrals
Aid in Identifying:
 under or over-utilized services
 providers who are not referring patients
 access or quality issues that impede service
utilization
 linkages between services that are not sufficiently
established
Aid in planning, resource allocation

26. Future Directions
Increase evidence that these tools:
 Help in referral strengthening
 Impact client outcomes
Better understand how they can be used in
different contexts

27. Integrating Health
Information Systems

28. Integration and Interoperability
of Health Information Systems
 Integration = combining two (or more)
different systems to create one system
 Interoperability = making two (or more)
different systems work together to give
the appearance of integration

30. Integration of Information Systems
 Information systems
 Data elements
 Indicators
 Data collection tools
 Reporting protocols, procedures
 Harmonization, rationalization of data
and indicators
 Data use

31. Interoperability (1)
 Horizontal – between different systems
at the same level
 Vertical – between sub-units of the system at
different levels of the health system
 Semantic – do the terms we use mean the same
thing?
 Vocabularies
o E.g. LOINC, SNOMED, HL7, ICD10
 Syntactic – what language are we speaking?
 E.g. XML, SDMX-HD

32. CHW in the VillageLocal Clinic Community Hospital
Clinical
Record
System
Rapid
SMS
Hospital
Record
System
Shared Record
Coordinated service delivery
Two-way information flow
Continuity of person-centred care
Source: Open HIE

33. Source: Open HIE

34. Integration of IS at
Community Level
 MEASURE Evaluation
HAITI
 CBIS 2006-2010
o Identified landscape of interventions and actors
o Identified needs of information
o Harmonized/rationalized data and indicators
o Harmonized data collection tools, reporting
protocols
o Monitored and supported implementation through
supervision/capacity building
o CLPIR toolkit

35. Integration at
Facility Level
 Beneficiary management
 Linking services
 Integrating data collectiontools, reportingforms
 Master client Index (unique id)
 Electronic patient recordssystems
(e.g. OpenMRS)
 Links to otherelectronicsystems(e.g. HR management)
on client ID
 Example (WHO/MEASUREEvaluation-3ILPMS)

37. Integration/Interoper-
ability at District Level
 Facility and system
management
 Data warehouse
 Master facility list with attribute data
 Examples:
o MEASURE Evaluation Ethiopia (SNNPR) HMIS
Scale-up
o MEASURE Evaluation Cote d’Ivoire – integration of
HIV/AIDS IS into RHIS

38. Integration/Interoper-
ability at National Level
 Policy development
and Planning
 Data warehouse
 Monitoring and evaluation
o E.g. MDGs
 HMIS governance
 Coordination of donors and other stakeholders
 Local IS enterprise architects
 Sustainable, scalable, incremental implementation
 Example: RHIS Data management standards
on Integration/interoperability

39. Wrap Up

40. Key Messages
 Integration can operate at all levels of the
health system and can include
interventions to all building blocks of the
health system.
 Integration can take many forms and is
inherently country-owned, country-led,
and context specific

41. Key Messages
 Despite the variability in integration models,
however, there are standardized tools,
approaches, and techniques that can be applied
to integrated health and development
 MEASURE Evaluation has worked to identify and
develop these resources, including pioneering
efforts to develop framework, indicators, tools,
and systems related to M&E of integration.

42.  Continue to develop, apply, and refine
resources intended to help countries to
design, implement, and evaluate
integrated health interventions, including
integrated service delivery and integrated
health information systems
Future Directions

43.  What are the pressing needs in M&E of
integration right now?
 Where should M&E of integrated health
interventions be going?

44. Resources
For links to resources and references relevant
to this presentation (including MEASURE
Evaluation and Non-MEASURE Evaluation
resources) see:
www.measureevaluation.org/eop/session-vi