Recomendados
Recomendados
Más contenido relacionado
La actualidad más candente
La actualidad más candente (20)
Destacado
Destacado (9)
Similar a Family planning 170706
Similar a Family planning 170706 (20)
Family planning 170706
- 1. Monitoring and Evaluation: FAMILY PLANNING PROGRAMS
- 2. Session Objectives • Be able to apply basic M&E concepts (frameworks, indicators, etc.) to family-planning programs • Be able to summarize the main issues in M&E of family-planning programs from a post-Cairo perspective. • Be able to summarize the emerging issues for M&E of family-planning programs in high HIV prevalence countries.
- 3. Session Overview • • • • • • Family-planning frameworks M&E implications of the Cairo agenda Contraceptive prevalence and unmet need Monitoring quality of care Evaluating the impact of quality Family planning and HIV
- 5. Conceptual Framework for FP Demand and Program Impact on Fertility Other intermediate variables Societal & individual factors Value & demand for children Fertility FP demand • Wanted • Spacing • Limiting Contraceptive practice • Unwanted Service outputs: Development programs • Access FP supply factors Service Utilization • Quality • Acceptability Source: Bertrand, Magnani, and Rutenberg, 1996. Other health & social improvements
- 6. Conceptual framework of family planning supply factors External Development Assistance FP Organizational Structure • Sectoral integration • Political support • Public-private partnerships • Delivery strategies • Management & supervision Service Outputs • Training • Service infrastructure Political and Administrative System • Access • Commodity acquisition & distribution • Quality • Acceptability • IEC • Research & evaluation • Resource allocations • Legal code / regulations Operations Larger societal & political factors Source: Bertrand, Magnani, and Rutenberg 1996
- 7. Applying the frameworks for FP M&E • Inputs, e.g. – Types and levels of resources – Qualified personnel – Unit and total costs of program resources • Outputs – functional areas, e.g. – People trained – Performance of people trained – Cost per person trained
- 8. Applying the frameworks for FP M&E • Outputs – Service outputs, e.g., – Service delivery points providing FP services – Quality of FP services – Cost of increasing access/quality of FP services • Outputs – Service utilization – New FP acceptors, Couple Years of Protection (CYP) – Returning clients – Cost of increasing CYP, etc.
- 9. Applying the frameworks for FP M&E • Outcome – intermediate outcomes – Contraceptive prevalence rate (CPR) – Unmet need – Costs associated with increased CPR • Outcome – long term outcome – Fertility rates – Unintended pregnancy – Costs of changes in fertility, unintended pregnancy
- 10. Indicators for FP programs • See Bertrand and Escudero, 2002, Compendium of Indicators for Evaluating Reproductive Health Programs, 2 volumes – Indicators that crosscut program areas – Indicators for specific program areas
- 11. What is different about M&E of FP programs? • Basic principles are the same as in other health programs • Outcomes relatively well-defined, focused, and measurable • Long history of data collection on FP outcomes through WFS, DHS – document global trends • Attempts to link outcomes to program outputs - evidence of program effects
- 12. Programme of Action adopted at ICPD, Cairo 1994
- 13. Traditional (pre-Cairo) focus of FP program M&E • • • • • • Demographic impact Focus on married women Availability of services Contraceptive adoption (new users) Characteristics of women Cross-sectional measurement
- 14. Cairo: Objectives of FP Programs • To help couples and individuals meet their reproductive goals • To prevent unwanted and high-risk pregnancies • To make quality FP services affordable, acceptable, and accessible • To improve the quality of family planning IEC, counseling, and services • To increase the participation and sharing of responsibility of men in FP • To promote breastfeeding to enhance birth spacing
- 15. Exercise 1 • Discuss the implications of the Cairo programme of action for M&E of FP programs. Identify 3 or more ways in which the traditional focus of FP programs listed on the earlier slide should change to respond to the Cairo agenda. What are the implications of these changes for M&E?
- 16. Contraceptive Prevalence Rate (CPR) • Percentage of (married) women of reproductive age (15-49) who are currently using a contraceptive method.
- 17. Unmet Need for Family Planning • Percentage of fecund women exposed to the risk of pregnancy who say they want to wait at least two years for another birth (spacing) or do not want any more children (limiting), but are not currently using a method of contraception.
- 18. Related Indicators • Demand for FP = % (married) women using FP + % (married) women with unmet need for FP • Percentage of demand satisfied = % (married) women using FP / % (married) women with demand for FP
- 20. CPR vs Unmet Need CPR • Relatively simple to define • Uni-dimensional • Consistency over time • Does not capture concept of meeting needs Unmet Need • Relatively complex to define • Multi-dimensional – demand & use • Definition has evolved • Captures concept of meeting need
- 21. Monitoring Quality of Care
- 22. What is Quality of Care in FP? • General, loosely-defined concept • Different people define quality in different ways • Multi-dimensional • Appropriate standards against which to measure quality vary
- 23. Bruce-Jain Framework • Choice of contraceptive methods • Information given to users • Provider competence • client/provider relations • re-contact and follow-up mechanisms • appropriate constellation of services
- 24. Indicators for QOC • No single indicator can capture the different components of QOC • Indicators need to be adapted to specific program context and priorities • Shortlist of 24 QOC indicators (see Bertrand and Sullivan, Evaluation Bulletin No. 1, Table 1 page 2).
- 25. Facility Surveys for QOC Indicators • Situation Analysis • MEASURE Evaluation Quick Investigation of Quality (QIQ) • MEASURE DHS+ Service Provision Assessments (SPA) • DHS service availability modules and community surveys (SAM)
- 26. Some Data Collection Issues • Small sample sizes for FP clients, especially in low prevalence countries • Observation in clinics that use a client flow approach • Sampling • Courtesy bias and hawthorn effects • Unit of analysis (client, provider, facility)
- 27. Case Study: QOC in Turkey
- 28. Turkey’s Strategic Framework S t r a t e g ic O b je c t iv e I n c r e a s e d u t iliz a t io n o f F P / R H s e r v ic e s I n t e r m e d ia t e R e s u lt 1 S t r e n g t h e n e d s u s t a in a b ilit y o f F P /R H p ro g ra m I n t e r m e d ia t e R e s u lt 2 E x p a n s io n o f h ig h q u a lit y F P / R H s e r v ic e s in t h e p u b lic a n d p r iv a t e s e c t o r s
- 29. The Quality Index • • • • • • Method availability Availability of trained personnel Perceived quality of FP counseling Adequate infection-prevention measures Availability of IEC materials Physical access to FP services
- 30. Data Source • Istanbul Quality Surveys – Facility inventory – Client exit interviews • Based on MEASURE Evaluation QIQ
- 31. The Quality Index • Sum of scores from the 6 components (range 0-6) Private hospitals SSK hospitals Health centers MCH/FP centers MOH hospitals 0 2 4 6
- 32. Method Availability • Proportion of facilities that distribute or prescribe 3 or more modern FP methods Private hospitals SSK hospitals Health centers MCH/FP centers MOH hospitals 0 0.2 0.4 0.6 0.8 1
- 33. Perceived Quality of FP Counseling • Proportion of clients who report – they were seated – had sufficient time with the provider – clearly understood the information provided Private hospitals SSK hospitals Health centers MCH/FP centers MOH hospitals 0 0.2 0.4 0.6 0.8 1
- 34. Adequate Infection Prevention Measures • Proportion of facilities that meet the following standards : – Plastic bucket for CL solution – Unused IUD kits kept sterile – Medical waste kept in leak-proof containers with lids – Appropriate containers for sharp objects Private hospitals SSK hospitals Health centers MCH/FP centers MOH hospitals 0 0.2 0.4 0.6 0.8 1
- 35. Evaluating the impact of quality of care
- 36. Framework for links between quality of family planning services and fertility Quality of services •Choice •Information to users •Provider competence •Client-provider relations •Follow-up •Appropriate constellation of services Other factors Acceptance Contraceptive prevalence Continuation Fertility Other proximate determinants Known effects Hypothesized effects Source: Jain, 1989
- 37. Outcomes of interest • Intention to use • Contraceptive adoption • Contraceptive discontinuation – Failure – Switching – Stopping • Current contraceptive use – Contraceptive choice • Unwanted pregnancy
- 38. Examples of impact studies • Peru (Mensch, et al., 1996) • Morocco (Steele, et al., 1999) • Bangladesh (Koenig et al., 1997)
- 39. Morocco Study Design (1) • To explore whether the service environment in which a woman resides affects adoption and continuation of the pill • Linkage of 1995 Demographic and Health Survey calendar with 1992 DHS Service Availability Module • Multi-level hazards models with contraceptive adoption and discontinuation as outcomes • 862 births and 775 episodes of pill use in 107 clusters
- 40. Morocco Study Design (2) • Explanatory factors - Individual and Community – age, education, residence, community drinking water & toilet facilities, principle economic activity – Contraceptive intention (discontinuation) – Breastfeeding status, last child wanted (adoption) • Explanatory factors – Program – Public health center <10km, pharmacy <5km, outreach services, 3+ methods available at clinic – Source of pills (discontinuation)
- 41. Predicted percentage of women adopting a modern contraceptive method within 12 months of giving birth by service factors 70 60 50 % 40 30 20 10 0 Yes No Health center <10km <3 3+ No. methods offered at closest set of facilities
- 42. Predicted 12-month pill discontinuation rate by reason and service factors, Morocco 35% 30% 25% 20% 15% 10% 5% 0% Government Pharmacy or other Source Failure Desired pregnancy Yes No Health center within 10KM Side effects/ health concerns Yes No Pharmacy within 5KM Other method-related
- 43. Main Findings: Morocco • Relatively strong service effects on postpartum adoption • Service availability associated with both adoption and discontinuation • Number of methods available only associated with adoption • Users of government sources have lower discontinuation
- 44. Limitations of Impact Studies • Measures of quality inadequate (often limited to access and method choice) • Cross-sectional designs (endogenous inputs) • Linking individual and program data (geographic boundaries, service environment vs. individual service experience)
- 45. Emerging areas: FP/HIV linkages and integration
- 46. Context • Considerable progress in preventing unwanted pregnancy but unmet need remains substantial • Rapid increases in HIV in many countries • Changing funding focus to HIV from FP • Integrated vs. vertical programs
- 47. Synergies between FP and HIV programs • Both are central to reproductive health • “ABC” messages in HIV programs also relevant to FP programs • Youth programs that encourage responsible sexual behavior prevent both HIV and teen pregnancy • Strong RH policies support both HIV and FP programs
- 48. Dual Protection • Abstinence • Monogamous couples using effective contraception • Correct and consistent condom use
- 49. FP in high HIV-prevalence countries • Relationship between HIV and fertility desires • FP/RH needs differ for: – – – – HIV- concordant monogamous couple HIV- concordant non-monogamous couples HIV discordant couples HIV+ concordant couples • HIV counseling in FP services
- 50. FP and VCT • FP counseling opportunity for VCT or general HIV counseling and VCT referral • VCT services could include FP services or FP counseling and referral • Concern over unintended consequences of integration – Provider burn-out – Discourage FP clients – Quality of integrated vs. vertical FP & VCT services
- 51. FP and PMTCT • Averts child infections by preventing unintended pregnancies among HIV+ women • PMTCT programs provide opportunity for prenatal FP counseling and post-partum contraceptive use • Reduced breastfeeding by HIV+ mothers will lead to shorter birth intervals in the absence of FP
- 52. FP Counseling of PMTCT clients, Zambia 60 50 40 % 30 20 10 0 ANC visit 3 months PP pregnant women HIV+ 6 months PP HIV- Source: Rutenberg & Baek, 2004
- 53. PMTCT-Client FP Use 6 Months Postpartum, Zambia 45 40 35 30 25 % 20 15 10 5 0 Modern Condom HIV+ Sex active, no method HIV- Source: Rutenberg & Baek, 2004
- 54. % FP sessions HIV Counseling in FP Sessions, Uganda 40 35 30 25 20 15 10 5 0 HIV needs assessed HIV risk discussed Dual protection proposed Baseline Follow-up Source: Rutenberg & Baek, 2004
- 55. Exercise 3 • Select an area of FP/HIV integration (e.g. PMTCT, VCT, HIV counseling in FP etc.). – Develop a basic input-output-outcome-impact framework for a simple program in this area. – Suggest 3-6 indicators to monitor your program. – What data sources would you propose to collect these indicators?
Notas del editor
- This module covers the M&E of family planning programs.
- By the end of this sessions you should be able to: Apply basic M&E concepts (frameworks, indicators, etc.) to family planning programs. Summarize the main issues in M&E of family planning programs from a post-Cairo perspective. Summarize the emerging issues for M&E of family planning programs in high HIV-prevalence countries.
- The session covers a number of topics as shown on this slide. We will begin with an overview of a general framework for the M&E of family-planning programs. We will then discuss the changes in family-planning programs since the 1994 International Conference on Population and Development (The Cairo Agenda) and discuss the M&E implications of the Cairo agenda through an exercise. We will then review two important indicators used in family planning programs; contraceptive prevalence and unmet need, again through an exercise. We will then go through an example of applying some of the general M&E concepts that have been introduced so far in this workshop in the context of monitoring the quality of care and evaluating its impact. Finally, we will turn to an emerging area; family planning in the context of HIV and review some of the data and indicators that apply in that context.
- So, we will first examine a common framework used for family planning programs.
- This slide shows a general conceptual framework for family-planning programs. The main impact shown on the extreme right of the framework is fertility, classified as wanted and unwanted. Family planning programs can have other wider health and social impacts too; for example increased family planning is likely to be associated with reduced maternal mortality due to fewer births and reduced infant mortality due to fewer high risk births such as high-parity births and births following short birth intervals. The intermediate factors (or outcomes) that directly affect fertility are contraceptive use (the subject of family planning programs) and other intermediate variables such as marriage patterns, breastfeeding, and abortion. Contraceptive use (and to a lesser extent, other intermediate variables) are in turn influenced by the demand for family planning to space and limit births and use of family planning services. Family planning demand is influenced by demand for children and by wider socio-economic factors, while service utilization is influenced by family-planning demand itself and service outputs such as accessibility, quality, and their acceptability. The service outputs themselves are influenced by family-planning supply factors and wider development programs. The area of the framework shown in blue is the area signifies the main functions of most family-planning programs. This conceptual framework shows how these functions link to outcomes and impacts at the population level and how they interact with wider social factors.
- This framework expands the blue boxes in the previous framework to provide a more detailed framework for M&E of family-planning program implementation. The intermediate outcomes shown on the right of the framework are the service outputs from the previous framework. These outputs are influences by the program operations such as management and supervision, training etc., which in turn are influenced by the organizational structure of the program. These form the program process. The political and administrative system, including the resource inputs to the program, influence these processes. Often programs will attempt to influence these factors too, for example by working to change the policy and regulatory environment to be more supportive of family planning so these can be inputs and processes.
- We will now turn to how to relate these conceptual frameworks to the classic input-output-outcome-impact M&E framework. The kinds of inputs to family planning programs are very similar to other programs, e.g. types and levels of resources, qualified personnel, and their associated costs. Outputs for family planning programs cover the different functional areas and are also similar to other programs. For example, for the functional area of training, outputs of interest would include people trained in family-planning activities, their performance, and the cost per person trained.
- Outputs can be at different levels. Service outputs for family planning program M&E include things like service delivery points providing family planning services, the quality of family-planning services, and the cost of increasing access and quality of family-planning services. Service utilization outputs are closer to the population outcomes and impact and include measures of the volume and cost of services provided such as new family-planning acceptors, couple years of protection (which accounts for the length of protection from pregnancy provided by different methods), returning clients, and the cost of increasing these various service-utilization outputs.
- Finally we turn to intermediate outcomes and long-term outcomes and impacts. For family planning programs these typically include the contraceptive prevalence rate, unmet need (which we will discuss in more detail later in this session) and fertility rates, particularly unintended fertility rates, as seen in first conceptual framework.
- There are many well-tested indicators for family planning and reproductive health programs. It is beyond the scope of this session to discuss these all in detail (and it would be very boring) but there is a comprehensive compendium of these indicators available from the MEASURE Evaluation project. We refer you to that for details of individual indicators. The compendium is divided into two volumes; indicators that crosscut program areas and indicators for specific program areas. A copy of the compendium is included on the CD of MEASURE Evaluation publications you received (note to speakers: confirm that they have the CD with the compendium).
- So, what is different about the M&E of family planning programs compared to M&E of other programs? Well, the answer in many ways is “not much”. The fundamental M&E principles discussed so far in this workshop (i.e. frameworks, indicators, data sources etc.) apply to family planning programs. However, family-planning programs have a few specific features. First, the outcomes are relatively well-defined, focused, and measurable, unlike in some other health program areas, including wider reproductive health. There is also a long history of data collection on family-planning outcomes through global-survey programs such as the WFS and DHS etc. This means that the data collection methods and indicators have been very well-tested and are pretty standard now and we have extensive documentation of global trends in these indicators going back to the 1970s and 1980s in many (but certainly not all) countries. Also, given historical population debates, there have also been several attempts to demonstrate whether or not family-planning programs work, giving rise to a relatively rich literature on family-planning program impact evaluation and associated methods.
- Family-planning programs have been in existence for many years, but in 1994 the International Conference on Population and Development in Cairo lead to dramatic shift in the focus of family planning programs that has implications for M&E.
- Before Cairo, family-planning programs were primarily (but not exclusively) motivated by concerns over population growth and the possible negative implications of population growth for development. Consequently programs focused on demographic impacts such as fertility rates and population growth rates and focused primarily on married women, since that was where most fertility occurred. When most programs began, few services were available, and few women had ever used contraception, so programs focused on making family-planning services more widely available and on contraceptive adoption by new users. Programs were supply-driven; i.e. the theory was that if you made family planning services sufficiently widely available women would use them, although they were also typically supported by demand-generation activities promoting the benefits of smaller families. Given the focus on married women, data collection and analysis tended to focus on the characteristics of women, and there was an emphasis on cross-sectional measurement of impacts through the large survey programs such as WFS, and later DHS. Family-planning programs certainly evolved as they matured and this general characterization became less true over time but it serves a general model of traditional, pre-Cairo family planning programs here.
- The ICPD in Cairo emphasized family planning in the context of a broader definition of reproductive health and included an explicit focus on gender equity and human rights. There was a very explicit move away from demographic motivations for family planning and demographic targets. The specific objectives that came out of the Cairo ICPD that are associated with family planning include: To help couples and individuals meet their reproductive goals To prevent unwanted and high-risk pregnancies To make quality FP services affordable, acceptable, and accessible To improve the quality of family planning IEC, counselling and services To increase the participation and sharing of responsibility of men in FP To promote breastfeeding to enhance birth spacing
- We will now break out into small groups to discuss the implications of these global shifts in policy related to family planning programs for the M&E of these programs. I would like you to break into small groups. (note to facilitator: determine how groups should form and provide relevant instructions. Suggested group size is 5-8). In your groups, discuss the implications of the Cairo programme of action for M&E of family planning programs. Think about the traditional focus of family-planning programs on the earlier slide and the kinds of measurement and M&E issues they would raise. Then identify at least three ways that those traditional areas of focus would change in response to the objectives declared in Cairo. What are the measurement and M&E issues associated with the changes in focus you identify? You will have xx minutes to discuss this in your group. Then we will review what each group came up with, so identify a person to report back on your discussions to the wider group (note to facilitator – insert length of time for discussion – suggest at least 15-20 minutes, with similar amount of time for discussion of report back).
- I now want to turn to a couple of particularly important outcome indicators for family-planning programs. The first of these is the contraceptive prevalence rate, defined as the percentage of women of reproductive age (usually defined as 15-49) who are currently using a contraceptive method. The CPR can be defined for all women, all sexually active women, or all married women. This is a traditional indicator for family-planning programs.
- Unmet need for family planning is defined as the percentage of fecund women exposed to the risk of pregnancy who say that they want to wait at least two years for another birth or do not want any more children but are not currently using a method of contraception. It aims to measure the gap between women’s stated fertility desires and their contraceptive behavior. It is a key indicator, particularly post-Cairo, because reducing unmet need is consistent with helping women meet their reproductive goals. The terms in turquoise need to be operationalized, but there are standard ways of defining them in DHS surveys that we won’t go into in detail here (but see the compendium of indicators or DHS recode file documentation for standard operational definitions of these terms).
- Two related indicators are the total demand for family planning, defined as the percentage of women using family planning plus the percentage of women with unmet need for family planning. This indicator combines both met and unmet need for family planning. The second relevant indicator is the percentage of demand satisfied, defined as the percentage of women using family planning divided by the percentage of women with demand for family planning (i.e. CPR/Demand). These indicators can be defined for all women, all sexually active women, or for married women, though of course it is important to ensure that the base population is the same when comparing them.
- For the second exercise, I would like you to return to the groups you were in for exercise 1. You have a series of graphs showing the trends in these four key family-planning outcome indicators for four countries. In your groups, discuss the trends in each indicator for each country, considering the following questions: Do the indicators show the same patterns in each country? Which countries seem to be the most successful in family planning based on each indicator? Do your conclusions vary depending on which indicator you look at? How do the trends in one indicator influence your interpretation or understanding of trends in the other? What does this tell you about the advantages and disadvantages of each indicator? You have about xx minutes for discussion and then we will come back to discuss your conclusions. Again, please identify a person from your group to report back on your discussions (not the same person as last time). Note to facilitator: Insert amount of time for discussion. Suggest a minimum of 15-20 minutes with similar time for discussion of report back, depending on the number of groups.
- Note to facilitator: The points on this slide can be incorporated into the discussion around exercise 2 instead of presented as a separate slide. This slide summarizes the main differences between the two key indicators of CPR and unmet need. CPR is relatively simple to define, is uni-dimensional (i.e. it measure only one thing, contraceptive use), is typically consistently defined over time and across studies, but does not capture the post-Cairo perspective of meeting needs. High levels of CPR would not be good if they reflect coercive use of contraception. In contrast, unmet need captures the concept of meeting expressed demand for family planning but it is relatively complex to define and calculate, is multi-dimensional in that it is influenced by both demand for and use of family planning, and its operational definition has evolved over time. Therefore, when looking at trends in unmet need it is important to be aware of any changes in the operational definition and their implications for trend analysis. Both indicators have their strengths and weaknesses, and both have an important role in family planning M&E. It is usually useful to examine both indicators.
- We will now turn attention to a specific example in family planning program M&E; monitoring quality of care. As already discussed, improved quality of family planning services is a stated objective of the Cairo programme of action. Therefore, defining indicators of quality and monitoring quality over time have become an important component of family planning program M&E.
- The first step is to define what is meant by quality of care in family planning. This is not as easy as it might at first seem. Quality in any program area, including family planning, is generally a loosely defined concept that will be defined in different ways by different people. It is also a multi-dimensional concept in that there are different aspects of quality that together define quality care. In addition, the appropriate standards against which to measure quality will vary across contexts according to local standards and guidelines, resource base, level of service provision, client expectations etc.
- The fundamental framework for defining quality in family planning programs was developed by Judith Bruce and Anrudh Jain. It defines six dimensions of quality that taken together define the quality of family planning services. These six dimensions are: the choice of methods offered, the information provided to the user, the clinical competence of the provider, the client/provider relation in terms of the interpersonal exchange (e.g. treating the client politely), re-contact and follow-up mechanisms, and appropriate constellation of services. The last of these, appropriate constellation of services, refers to locating an appropriate mix of services that are physically accessible to the client, and open at convenient times. Measurement of quality should ideally address all six of these dimensions.
- Operationalizing these six dimensions of quality into measurable indicators that meet indicator standards is quite challenging. No single indicator can capture the different components of quality in a meaningful way given the multidimensional nature of quality described by the Bruce-Jain framework. One challenge is to avoid a proliferation of indicators to measure every detailed dimension of quality leading to an unwieldy and impractical number of indicators to deal with. In addition, given that the standards against which quality should be defined vary by context, indicators of quality typically need to be adapted to specific program context and priorities. For example, indicators of provider competence need to be based on local clinical practice standards, and what is expected in a rural health post will not be the same as what is expected in a university hospital. Standard indicators of quality can be defined in broad terms but the details have to be operationalized differently in different settings making comparisons across contexts difficult. Based on expert opinion a suggested minimum set of 24 indicators for quality of care have been published by the MEASURE Evaluation project. Again, it is beyond the scope of this session to review these indicators in detail but you are referred to the publication listed on this slide or to the QIQ manual published by MEASURE Evaluation, both of which are available on the CD of MEASURE Evaluation publications that you have. (Facilitator note: confirm participants have the CD of publications).
- The most common source of data for quality of care indicators are facility surveys. Examples of standardized facility survey instruments that can be used to collect indicators of quality of care in family planning include the Situation Analysis tool developed by the Population Council. This was the first tool to attempt to systematically measure quality of care and focused only on family planning. The MEASURE Evaluation Quick Investigation of Quality grew out of the Situation Analysis but was designed to be a low cost, practical tool to measure the minimum set of 24 indicators of family planning service quality mentioned on the previous slide. However, it still includes a facility inventory, client/provider observation module, and client exit interview in order to capture the different dimensions of quality. The MEASURE DHS SPA is a more extensive tool that aims to measure indicators or quality, defined as readiness to provide services, for a number of service areas including family planning. The QIQ questions were included within the SPA family planning modules. The SPA includes a health worker interview as well as the components mentioned for the QIQ. Early DHS service availability modules and community surveys aimed to measure physical access to family planning services. Some of these included visits to the nearest facility to a DHS sampling unit to collect basic information on the services provided using a facility inventory and thus provide limited data on some dimensions of quality of care (mostly methods available and physical access as a component of the appropriate constellation of services). The sampling for these surveys was very different from that of the other facility surveys described here because they were designed to provide population-based indicators of physical access rather than facility-based indicators of quality. The SAM has been replaced now by the SPA.
- Collecting data on the quality of family-planning services raises some challenges. In populations in which contraceptive use is low (most of rural Africa, for example) the number of family-planning clients that visit a clinic while an interview team are present will be small; in many clinics no family planning clients will come during the interviewer visit. Therefore, the number of client/provider observations and client exit interviews will be low and skewed toward larger clinics in higher-prevalence areas. One option is for the interview team to stay in low-volume clinics for several days to increase the chance of observing a family-planning consultation and interviewing the client, but this is not very cost-effective. Therefore client-based indicators of quality of family-planning care will often be based on small sample sizes and will not be amenable to further breakdown by client or provider characteristics or be suitable for detecting small changes over time. Some clinics use a client flow approach where clients first see a lower-level health professional for basic triage and counseling before seeing a higher-level professional for the actual provision of a family-planning method. If the observation team follows the higher-level provider rather than the client, key elements of care that are provided during triage may be missed and quality under-estimated. This needs to be considered when designing the survey implementation protocol. Sampling for facility surveys can be challenging as sampling frames are often incomplete or out-of-date. In addition, a decision has to be made as to whether to conduct a stand-alone facility survey or one linked to a household survey, as described earlier in this workshop (facilitator note: confirm earlier modules have addressed different sampling designs for facility surveys). Another challenge relates to the accuracy of responses from clients and from observations. Clients will often report higher levels of satisfaction that they actually experience because of a desire to please the interviewer. Similarly, providers are likely to perform better than usual when being observed (hawthorn effect). Some research has been done on these issues; indicators of quality that are sensitive to these types of biases should be considered over-estimates of the typical quality. Finally, there are several units of analysis for indicators of quality of care; the facility, the provider, the consultation, and the client. The facilities will be selected according to a pre-determined sampling design but the observations and clients are essentially a convenience sample of all, or a sub-sample of, clients who show up on the day of the survey. This has implications for the types of biases that are potentially present in the data on different units.
- Lets now examine a case study of measuring quality of care in the context of a specific family-planning program in Turkey. (Facilitator note: could change this to Country A or replace with a regional example).
- This is the Results Framework for a family-planning project in Turkey. The overall objective of the program is to increase utilization of family planning and reproductive health services. Indicators at this level included the CPR and the all-method contraceptive discontinuation rate collected from household surveys. The two intermediate results relate to strengthening the sustainability of FP/RH programs and expanding high-quality FP/RH services in the public and private sectors. Monitoring this second result requires appropriate indicators of the quality of family-planning services.
- In order to monitor the second result, the Turkey family-planning program developed a composite indicator based on six dimensions of quality; method availability, availability of trained personnel, perceived quality of FP counseling by clients, adequate infection prevention measure, availability of IEC materials, and physical access to services. These do not map exactly onto the six elements of the Bruce-Jain framework but broadly cover 5 of the 6 elements (follow-up and re-contact mechanisms are not included) and represent the main areas of the family planning program directed at improving quality. Note how this indicator is multi-dimensional and is specific to the context of the program.
- The data source for the quality index was a facility survey conducted in Istanbul, the area in which the programs were most active. The Istanbul Quality Survey was based on the MEASURE Evaluation QIQ tool but adapted to meet the program’s own defined measure of quality. The survey included a facility inventory and client exit interviews but no observation. The observation module adds considerably to the cost and complexity of the facility survey implementation and the Istanbul facility survey was designed to be a fairly rapid, low-cost exercise. However, the lack of an observation module means that there is no direct expert assessment of provider competence, so the program has to rely on proxy indicators of provider competence, such as presence of trained providers and infection prevention standards. The survey covered all higher-level facilities and a 50% sample of health centers. It included public and private facilities, consistent with the result it is associated with which specifies improving quality FP/RH services in the public and private sectors.
- This chart shows the overall scores on the Quality Index by type of facility. The Quality Index is composed of the sum of scores on six binary (i.e. 0/1) indicators of quality covering the six dimensions described earlier. Scores range from 0 to 6, with 0 indicating low quality (0 score on all six component indicators) and 6 indicating high quality (1 score on all six component indicators). Overall health centers score lowest, with an average score of just under 2, and MCH/FP centers score highest, with an average score close to 4. While the Quality Index is a useful simple summary of overall quality for monitoring purposes, it is not very useful for program management because it does not give any indication of which dimensions of quality are weakest and need most program attention. For this reason it is usually most useful to look at the summary indicator in conjunction with its component indicators.
- This chart shows the method availability component of the Quality Index by facility type. The method-availability indicator is defined as 1 if a facility distributes or prescribes three or more modern family planning methods and 0 otherwise. The summary indicator presented here represents the proportion of facilities of each type that distribute or prescribe three or more methods. Private and SSK hospitals and MCH/FP centers all provide three or more methods, but only about 80% of MOH hospitals and 40% of health centers do so.
- The component indicator for perceived quality of FP counseling focuses on the interpersonal dimensions of the consultation. The indicator is defined at the client level. The indicator is defined as 1 if the client reports that they were seated, had sufficient time with the provider and clearly understood the information provided and 0 otherwise. Note that the client must report all three items to score 1. The indicator is then summarized across facilities as the proportion of clients in each type of facility that scored 1. Private hospitals scored best on this dimension of quality, with just over 80% of clients reporting satisfaction with FP counseling, while health centers and MOH hospitals scored lowest, with about 65% of clients reporting quality counseling as defined here.
- The indicator for adequate infection-prevention measures was defined at the facility level. A facility scored 1 on this indicator if it met each of the four standards listed: plastic bucket for chlorine solution, unused IUD kits kept sterile, medical waste kept in leak-proof containers with lids, and appropriate containers for sharp objects. Again, facilities had to meet all 4 standards to score 1, otherwise they scored 0. MCH/FP centers performed best on this dimension of quality; just over 40% met all 4 standards, while private hospitals scored lowest, with less than 10% meeting all 4 standards. At first glance, infection-prevention standards appear to be shockingly low in private hospitals. However, this illustrates some of the limitations and dangers in operationalizing the measurement of quality. The standards defined in the indicator appear reasonable from a clinical perspective. The survey protocol required interviewers to observe all 4 items to score a 1. However, in private hospitals there were a large number of consultation rooms and interviewers could not gain access to some of them to confirm whether they had appropriate containers for sharp objects and that medical waste was kept in leak proof containers with lids. Therefore, they marked the item as not seen and the facility scored 0. In addition, some private hospitals used alternative, but acceptable, sterilization systems to chlorine solution, yet because the standard was based on chlorine solution they scored 0 on this item. It is important to think about these kinds of issues when defining indicators and associated data collection protocols. However, sometimes these kinds of problems do not become apparent until data collection or analysis. It is useful to have protocols for dealing with such problems as early as possible if they are identified. These three component indicators illustrate how the overall Quality Index was constructed and illustrate some of the issues that arise in measuring quality. I will not review the remaining 3 components here. Further details of this study can be found in the QIQ Country Case Studies Technical Report on your CD of MEASURE Evaluation CDs. (Note to facilitator: confirm they have the CD mentioned).
- We will now turn to evaluating the impact of quality of care.
- This is a conceptual framework that specifies the links between the quality of family-planning services and outcomes and impacts at the population level developed by Anrudh Jain in 1989. On the left of the framework are the six dimensions of quality as defined in the Bruce-Jain framework. Quality of care is hypothesized to influence both contraceptive adoption and contraceptive continuation, which in turn influences contraceptive prevalence and fertility. There is also another “black box” for other factors that influence acceptance and continuation as well as other proximate determinants which in turn influence fertility. These boxes are not expanded further because they are not central to the conceptualization of the impact of quality on population outcomes. The impact evaluation question is, therefore, to demonstrate the link between quality of family planning services and population outcomes such as contraceptive adoption and continuation and current contraceptive use.
- Given the conceptual framework just described, likely outcomes of interest for impact evaluation include intention to use contraception, contraceptive adoption, contraceptive discontinuation (which can be further analyzed by the type of discontinuation), contraceptive use and method choice, and unwanted pregnancy. Of these outcomes, intention to use is a more intermediate outcome as it is a precursor to contraceptive adoption. Contraceptive adoption and discontinuation are closest to quality in the conceptual framework so is where we would expect to find the strongest impacts. The relationship between quality of care and contraceptive discontinuation is generally hypothesized to be stronger than the relationship with adoption Contraceptive use and unwanted pregnancy are more distal outcomes that are likely to be mediated by other factors resulting in weaker direct links with quality of care.
- There have been a number of attempts to evaluate the impact of quality of care in the literature. Three of these studies are listed here. The evaluation designs used in these studies varied and each had their strengths and limitations. The Peru study examined the influence of quality on current contraceptive use based on linked Situation Analysis and DHS data. The Morocco study examined the impact of quality on contraceptive adoption and discontinuation based on linked SAM and DHS data. The Bangladesh study examined the impact of perceived quality on contraceptive adoption and discontinuation based on longitudinal data collected as part of the Matlab Demographic Surveillance System and an associated survey of women on their perceptions of the quality of community family planning worker visits. Given time constraints, we will focus in more detail on the design of one of these studies, the Morocco study. This is not necessarily the best study design, but it serves as an illustration of some impact evaluation design issues. References for the other two studies are given in the reference list for this module. Note for facilitator: There is a more recent study from the Philippines by Ramarao et al from 2003 or 2004. This study should be added to the slide and speaker notes if you have access to it.
- The objective of the Morocco study was to evaluate whether the service environment in which a woman resides affected adoption and continuation of the pill. The study linked 1995 DHS calendar data with the 1992 DHS SAM. Two multilevel hazards models were fitted; one for contraceptive adoption and one for contraceptive discontinuation. The models were multilevel models to allow for the fact that quality is measured at the community level. There are also multiple observations per women because adoption refers to adoption following a birth so women can have multiple births and therefore adoptions within the 5 years covered by the calendar. Similarly, discontinuation is measured for each episode of use within the 5 years covered by the calendar, so women can have multiple episodes of use. The sample size for the contraceptive adoption model was 862 births and the sample size for the discontinuation model was 775 episodes of pill use. These occurred in 107 clusters, or communities.
- A number of explanatory factors were included in the model. At the individual and community levels these included age, education, contraceptive intention (i.e. spacing versus limiting) for the discontinuation model, breastfeeding status, and the wanted status of the last child for the adoption model. At the program level explanatory variables were whether or not there was health center within 10km, a pharmacy within 5km, outreach services to the community, and three or more methods available at the closest clinics. For discontinuation, the source of contraceptives was also included (government clinic, pharmacy). Note that these program variables only cover a very limited set of elements of the Bruce-Jain framework; essentially only physical access to services which can be considered part of the appropriate constellation of services and choice of methods available. To some extent the source of supply captures the quality environment in each type of facility, which is expected to differ substantially, but it does not provide any information on which dimensions of the service quality environment in different types of facility are important for influencing continuation.
- This graph shows the predicted percentage of women adopting a modern contraceptive within 12 months of giving birth by different service factors. Being within 10km of a health center increases the probability of accepting a modern method, as does having three or more methods available at the closest set of health facilities.
- &lt;number&gt; This slide shows the predicted probability of discontinuing pill use within 12 months by reason for discontinuation and service factors. Overall, the probability of discontinuing pill use is higher if pills were obtained from a pharmacy compared to a government health facility. This is true for all reasons for discontinuation, but especially for other method-related reasons, which includes cost, availability etc. The overall level of discontinuation was similar for those who lived within 10km of a health center and those that did not but the reasons for discontinuation differed significantly; those within 10km of a health center were more likely to discontinue due to side effects, while those further from a health center were more likely to experience a contraceptive failure. Finally, those who lived more than 5km from a pharmacy were more likely to discontinue than those within 5km of a pharmacy, particularly due to side effects and health concerns.
- To summarize, the study found relatively strong service effects on post-partum adoption of contraception and that service availability (i.e. physical access) was associated with both adoption and discontinuation of contraception. However, the number of methods available (method choice) was only associated with contraceptive adoption. The study also found that users of government sources have lower discontinuation rates than users of pharmacies, but it is not possible to determine what aspects of the service quality environment at government facilities rather than pharmacies influence discontinuation.
- Impact studies such as this have a number of limitations of course. In this case, the measure of quality used was very limited; it only covered two dimensions of the Bruce-Jain framework and missed elements such as the quality of the client-provider interaction, the information provided to the user, and re-contact and follow up mechanisms which are hypothesized to influence continuation. This is a reflection of the type of data available on quality that could be linked to population-based data on discontinuation at the time the study was conducted and more recent studies have used better measures of quality. The study uses a cross-sectional design, which means that there are a lot of endogenous inputs, or selection effects, and it is difficult to control fully for all of these. There are also issues to be considered when linking individual and program data, such as what are the appropriate geographic boundaries for linking services to population? In this case distance to different types of facilities was used, but this is rather arbitrary. Similarly, in this study characteristics of the general service environment were used, but this is not the same as individual service experience but it can be hard to link individuals to the facilities they use. Client follow-up studies and questions on perceived quality of care are ways of trying to deal with this issue.
- For the final part of this module we will discuss an emerging area in family planning monitoring and evaluation: family planning and HIV program linkages and integration. We will review the linkages between the two program areas and the potential for integrated family planning and HIV programs and then review some indicators that can be used to monitor the level of integration.
- We will begin with the family planning and HIV program context. Family-planning programs have been in existence for many years and considerable progress has been made in reducing unwanted pregnancy. Yet, unmet need for family planning remains high in many countries, especially in sub-Saharan Africa where fertility remains high. At the same time, there have been rapid increases in HIV prevalence in many countries, again particularly in sub-Saharan Africa. Global funding initiatives have shifted focus from family-planning programs to HIV programs, particularly in recent years, in response to this changing situation. Both family planning and HIV programs have been separate vertical programs, but there is potential for a more integrated approach.
- Integration of family planning and HIV programs only makes sense if there are synergies between them, i.e. it is mutually beneficial. There are a number of synergies between family planning and HIV programs. First, both are central to the wider concept of reproductive health contained within the Cairo programme of action. The Abstinence, Be Faithful, Correct and Consistent Condom Use messages in HIV prevention programs are also relevant for preventing unintended pregnancy (at least the A and C messages). And programs aimed at youth to encourage responsible sexual behavior help prevent both HIV infection and unintended pregnancy. Further, strong reproductive-health policies will support both family-planning and HIV-prevention objectives.
- There are a number of ways of protecting against both unintended pregnancy and HIV infection. These dual protection behaviors include abstinence, monogamous couples using effective contraception, and correct and consistent condom use.
- There are some specific issues that need to be addressed in family-planning programs in countries with high HIV prevalence. One of these is the relationship between HIV and fertility desires. We know relatively little about this relationship, which could operate in several ways. For example, couples living in high HIV prevalence areas may wish to have additional children rapidly while they are healthy and to insure against higher mortality levels among children in high HIV areas even if they do not know their individual HIV status. Women or couples who know or suspect they are HIV positive may wish to reduce fertility to avoid passing the virus on to their children and to avoid leaving additional orphans behind. These potential dynamics between individual HIV status, community HIV prevalence and fertility desires have important implications for family planning programs. The family planning and wider reproductive-health needs of couples differ according to whether they are concordant HIV-negative and monogamous, concordant HIV-negative and non-monogamous, HIV-discordant, or concordant HIV-positive. Family-planning consultations provide the opportunity to counsel on different options and to counsel on HIV as well as pregnancy prevention.
- Family-planning programs provide an opportunity for counseling and testing associated with HIV prevention. Similarly, HIV-prevention programs may provide synergistic opportunities for family-planning counseling. Individuals or women who attend voluntary counseling and testing services are clearly sexually active and may also be in need of family-planning services. Family-planning services or counseling and referral could be integrated into VCT services. However, there are some valid concerns over integration of VCT and family-planning services, including provider overload and burn out, given the very intense nature of VCT service provision. There are also concerns that integrating VCT into family planning services might create stigma and discourage family-planning clients, and concerns that the quality of both services might suffer due to work load if the services are integrated.
- Another HIV program area which has natural synergies with family-planning programs is prevention of maternal-to-child transmission (PMTCT). Family planning is a direct component of a comprehensive approach to PMTCT as described by the WHO, because it prevents unintended pregnancy in HIV-positive women, thereby avoiding vertical HIV transmission. PMTCT programs for pregnant women could provide opportunities to counsel on future family planning and to provide post-partum family-planning services in the context of post-partum care or post-partum ARV treatment in PMTCT+ programs. Post-partum family planning becomes particularly important if breastfeeding durations are decreased to reduce the risk of transmission through breast milk, which would lead to shorter birth intervals and higher risk of vertical transmission in the absence of family planning.
- The context described by the previous few slides provides the basis for developing integrated programs and associated M&E frameworks. The basic indicators to monitor family planning in the context of HIV will generally be very similar, but they will be used in conjunction with other indicators related to HIV or in relation to specific sub-populations. For example, this slide shows the percentage of PMTCT clients that received family-planning counseling at different points of their care by HIV status. Less than 40% of PMTCT clients received family planning counseling during ANC visits (before testing), but more than 50% received family planning counseling 3 months post-partum, and 40-50% received family planning counseling at 6 months post-partum. Family planning counseling was higher among HIV+ clients than among HIV- clients, especially at 6 months post-partum. The indicator of whether or not a client received family planning counseling is a fairly standard process indicator obtained from facility-survey observations or exit interviews, but the fact that it is specific to PMTCT clients allows us to monitor the extent of integration of family planning in PMTCT services.
- This slide examines family-planning use among PMTCT clients at 6 months post-partum by HIV status. This indicator represents an intermediate outcome for family-planning programs in the context of high HIV and an intermediate outcome for comprehensive PMTCT programs. More than 35% of both HIV-positive and HIV-negative clients were sexually active and not using a contraceptive method at 6 months post-partum, and a little over 30% of both HIV-positive and HIV negative clients were using a modern method. However, HIV-positive clients were much more likely to be using a condom at 6 months post-partum than HIV negative women.
- This last slide illustrates the use of indicators to monitor the integration of HIV-counseling into family-planning services (i.e. integration in the opposite direction). The chart shows the percentage of family planning sessions that include different aspects of HIV counseling before and after an intervention to integrate HIV counseling into family-planning services. The indicators demonstrate marked increases in the assessment of HIV needs, discussion of HIV risk, and promotion of dual protection in the context of family-planning sessions. The increase was particularly pronounced for counseling on dual protection.
- For this final exercise, divide up into groups again. Select an area of FP/HIV integration such as integrating family planning in PMTCT services, or into VCT services, or vice-versa. Develop a basic input-output-outcome-impact framework for a simple program in your selected program integration area. Suggest 3-6 indicators to monitor your program, and identify the data sources you might need to collect these indicators. You have about xx minutes for discussion and then we will come back to discuss your framework and indicators. Again, please identify a person from your group to report back on your discussions (someone who has not already reported back). Note to facilitator: Insert amount of time for discussion. Suggest a minimum of 60 minutes with at least 40 minutes for discussion of report back, depending on the number of groups.