1. プログラム評価セミナーⅠ アーバン・インスティテュート ( The Urban Institute ) 開発・原作 ： レイモンドＪ．ストライク ( Directed by Raymond J. Struyk ) 構成・提供 ： 上野真城子 ( Presented by Makiko Ueno )
12. プログラム評価セミナーⅠ アーバン・インスティテュート ( The Urban Institute ) 開発・原作 ： レイモンドＪ．ストライク ( Directed by Raymond J. Struyk ) 構成・提供 ： 上野真城子 ( Presented by Makiko Ueno )
24. クライアント（顧客利用者）の満足度 << プログラム評価② もっとも多く接触した担当職員の対応の評価 Vladimir 市 Gorodetz 市 Fully satisfied 全く満足 62 62 Satisfied 満足 23 27 More or less satisfied どちらともいえない 7 11 Not satisfied 不満 4 1 Not satisfied at all 全く不満 2 - No answer 2 -
32. プログラム評価セミナーⅠ アーバン・インスティテュート ( The Urban Institute ) 開発・原作 ： レイモンドＪ．ストライク ( Directed by Raymond J. Struyk ) 構成・提供 ： 上野真城子 ( Presented by Makiko Ueno )
42. プログラム評価セミナーⅠ アーバン・インスティテュート ( The Urban Institute ) 開発・原作 ： レイモンドＪ．ストライク ( Directed by Raymond J. Struyk ) 構成・提供 ： 上野真城子 ( Presented by Makiko Ueno )
56. プログラム評価セミナーⅠ アーバン・インスティテュート ( The Urban Institute ) 開発・原作 ： レイモンドＪ．ストライク ( Directed by Raymond J. Struyk ) 構成・提供 ： 上野真城子 ( Presented by Makiko Ueno )
Note to instructor: emphasize and discuss the items underscored. The key point is that evaluation requires making judgements about how well a program is working and/or achieving its intended results. As well will see, to make judgements requires that one has criteria against which to measure program performance.
P = process or implementation evaluation I = Impact evalution B-C = benefit- cost analysis In B-C analysis, ideally the benefit measure is the program impact. For example, in a training program the principal impact is increased earnings per year. The B-C might be the program cost to achieve the discounted present value of the increased earnings over a 10 year period.
Accountability : by knowing in detail how a program is working and what it is accomplishing it is possible to hold program administrators responsible for the program’s operation Funding : evaluation results very often have a strong impact on government and parliamentary decisions to expand, contract, or even eliminate funding for a program. Ownership : means staff idenitfying with the program and taking responsibility for it. Program leaders and staff often become more committed to seeing the program well-run by participating in the process of improving it. Policy impact: Results of well-executed evaluations are very difficult for senior administrators to ignore, especially if they have agreed to the questions being addressed in advance. Evaluations findings almost guarantee the evaluators a seat at the table when program improvements are being discussed.
Early evaluations in the U.S.: Pt. 1: By 1930s, social scientists in various disciplines were advocating the use of rigorous research methods to assess social programs. Evaluations became somewhat more frequent Study of introducing boiling water as public health practice in villages in Middle East was landmark study Pt. 2: Types of programs subjected to some type of evaluations by the 1950s Delinquency prevention programs Psychotherapeutic treatments Public housing programs Educational activities Community organization initiatives
Academic development First textbooks at the end of the 1970s Evaluation Review founded; others followed rapidly From the mid-1960s the national government in the U.S. routinely commissioned program evaluations. This spurred the development of the evaluation industry. On the supply-side were think tanks. Clients included the various Congressional offices as well as the ministries. From the early 1970s social experiments and piloting new programs were popular with Government agencies. These both required detailed and sophisticated evaluations.
The right way: Evaluation method should be driven by the questions being asked--not the other way around. The evaluator must build an audience among decision makers if the results are to be used. This step is very often not done. The evaluator assumes he knows which questions are of the greatest importance and interest and molds the evaluation around them. The result is that there is often little interest in the findings among the people actually responsible for the program.
Primary intended users : these are the people who can use the evaluation results to change the program Those funding the evaluation are often not the primary audience. For example, the World Bank may engage a firm to evaluate a social assistance program in India. But it is only the Indian officials that have the power to change the program. So it is essential to engage them in defining the issues to be addressed. Program administrators must be convinced to adopt changes. This is not just the highest officials. So try to engage representatives of the program’s field offices, for example. The opinions of Clients can be critical in defining an evaluation particularly where participation rates are low. What are they finding objectionable about the program? Once you have identified the right people, how do you engage them? For a single stakeholder start by asking: What you most want to know about the program? Name three pieces of information that would be at the top of your list.
Emphasize the following points in this definition: program performance program is functioning as intended according to some appropriate standard for two different kinds of issues: service utilization (issues of program participation) and program organization and operations (the actual operations of the program) This type of evaluation permits the program managers to understand how the program actually operates. Knowing this is often essential for interpreting the results of an impact evaluation and determining how to change the program to make it more effective.
Similar program experience -- across many programs targeted on the poor in the U.S. participation rates are 50-60% of those believed eligible to receive benefits. So if you are studying such a similar program, your views about reasonable participation rates should take this fact into account. To determine standards for administration practices look at the practices of programs widely regarded to be well operating; look at their practices in such areas as program documentation (e.g., guidelines for in-take workers), training, record keeping, quality control, and reporting.
Bias includes Errors of exclusions – leaving out those you want to serve Errors of inclusion – serving those the program is not intended to serve Target population awareness is especially important where participation rates are low.
Q1 concerns program coverage. Q2 address the targeting of benefits. Q3 addresses client satisfaction. This was especially important in 1994 because housing allowances were the first means-tested program introduced in Russia. The reference for this case study is: R. Struyk, L. Lee and A. Puzanov, &quot;Monitoring Russia's Experience with Housing Allowances,&quot;, Urban Studies , vol. 34, no.11, November 1997, pp. 1789-1818.
Pt. 1: In 1991 the State housing stock was transferred to the ownership of local governments, who were then completely responsible for its maintenance and operations. Pt. 2: The typical local government was spending half of its total budget to operate the housing stock, including associated subsidies for utilities Pt.3: In December 1992 a framework housing reform law as passed. It called for raising rents and utility fees on municipal housing to full cost recovery levels over a multi-year period. The timing of the increases was left to local governments because they owned the stock. The law specified that the process was to begin in January 1994. Rents could only be raised, however, if the housing allowance program was established in a city to protect poor renters.
S = the amount of the subsidy MSR = the “maximum social rent,” which is the cost estimated for a unit of a suitable size for the family applying for the subsidy. It is based on unit size (sq.m.) and standard use rates for utilities. A participant household can occupy a bigger unit or use more utility services but the subsidy is fixed for the standard unit and consumption. Y = household income t = the share of its own income the household must spend on housing before receiving the subsidy This is a gap formula because the subsidy fills the gap between the cost of a standard unit (MSR) and what the family can reasonably afford to pay (tY). Y* is the income cutoff, I.e., the lowest income a family can have and not receive a subsidy. Take some time with the students to think what happens to the number of households eligible for housing allowances as the values of MSR and t are increased and decreased.
The responsible ministry and the evaluation team wanted to get an early reading on how the program was working, so two cities than implemented rent increases and housing allowances in the spring of 1994 were selected for the evaluation. A survey of the population was required to estimate the share of eligible households who were participating. The additional sample of program participants was needed to insure that the sample for analyzing participant satisfaction with the program would be large enough. (If participation rates were low, then there would only be a few participants included in the general survey of the population. The evaluation was conducted in both 1994 and 1995 because of the findings in 1994.
1. The share of household eligible for payment changed a lot in Gorodetz between the two years. This is because the city raised the household contribution rate (t) a lot between the two years. It did this to control the cost of the allowance program. (In 1994, the household contribution rate was on a sliding scale with the poorest households only have to pay 2.5% of their incomes on rents.) 2. Participation in 1994 among eligible households was very low. This was a surprising finding. In part this is explained by the fact that rents remained low in absolute terms, so that the benefits of participating in the program were low. Nevertheless, the finding created great concern in the Ministry because it was feared that poor households would refuse to pay their rents or, worse, begin demonstrating against the rent increase. Based on these findings the minister sent a letter to every state governor urging him to press cities to conduct a large information campaign about the availability of the subsidy before the next rent increase. 3. Participation rates rose a lot by 1995, especially in Gorodetz where the information campaign was more effective. Participation was encouraged by a second round of rent increases which increased the value of the subsidies (MSR rose).
The figures show that benefits are very strongly concentrated on poor families. So the program is successful in this respect.
The satisfaction levels are indeed strikingly positive. Hence, income-testing was clearly not being rejected by the participants. Conclusions The program got off to a good start overall. Participation rates were and remain a problem. In 2001 about 30 percent of eligible households participated across the country. It has served as a model for other means-tested programs in the country.
Pt 1, example: for program outreach, is the expected volume of people applying for services? Pt 2: how long do clients have to wait for services? what are the error rates in processing them? How long are potholes on the streets before they are repaired? Is the quality of garbage pick-up consistent with the city’s standards?
The study that is the source for this example is, V. Gabor and C. Bsotko, Changes in Client Service in the Food Stamp Program After Welfare Reform . Washington, DC: Health Systems Research, Inc., report to the USDA Food and Nutrition Service, 2001.
Broad legislation was enacted in 1996 that fundamentally changed the structure of basic assistance for low income families. The “old program” was called, Aid to Families with Dependent Children. A female headed-family could receive benefits indefinitely under the program. And about 25% of those receiving benefits at any time were long-term (10+ years) recipients. The original program had very weak job-search requirements but these were gradually strengthened over time. The “new program” is called, Temporary Assistance for Needy Families (TANF). The Food Stamp and TANF program work closely together; generally, but not always, the two programs are administered from the same office. Under the Food Stamp program low-income households receive vouchers (that look like play money) that they can use to purchase food and certain other necessities at regular stores. Beneficiaries pay for the stamps, the amount they pay as a percent of the value of the stamps depends on their incomes--poorer families pay less. Very poor families pay about 20% of the value of the stamps; for example, they pay $40 and receive stamps worth $200.
Examples of service providers include firms conducting job training programs for clients telephone call centers that contact clients for the agency, for example, to remind them that it is time to recertify their eligibility to receive benefits.
Bullet #2: employment offices are generally located separately from the agency administering TANF and Food Stamps. So being sent to the Employment Office is typically a significant effort for the applicant, many of whom do not own a car. Bullet #3: Agencies “punish” clients for not fulfilling their obligations under the contract they sign by reducing or eliminating Food Stamp benefits for 3 or 6 months, for example. Results were used by both the responsible national ministry and the state agencies that administer the program to improve the flow of clients so that they have access to the Food Stamp benefit earlier in the process.
Outcome : Status of the social or physical condition the program is accountable for improving. Impact : The net effect of a program. The difference between outcome and impact is that an outcome is the status, which may or may be due to the program’s intervention. An impact is clearly linked to the program. The usual practice is to identify the impact by contrasting the outcomes for an experimental group that receives a treatment and a control group that does not. Example: The objective of the housing allowance program is to protect low income households from having to pay the full cost of housing. In this way participants after more money to spend on non housing consumption. Can the impact of the program being measured by knowing the size of the housing allowance a family receives? No. Consider the following case. An elderly widow receives income from her son to help her with expenses. When she begins receiving the housing allowance subsidy, she tells her son. He in turn reduces his monthly support by about half of the value of the housing subsidy. So the impact on this woman’s non housing consumption is only half of the value of the subsidy payment.
Selection bias : Errors made in assigning targets in random process or when “similar” controls are selected they were not matched with the treatment group on a critical factor. Secular drift : General increase in homeownership rate makes it hard to isolate the effect of a mortgage lending initiative. FNMA--the secondary mortgage market firm-- in late 1980s announced a major change in its underwriting standards to make it possible for families with lower incomes to purchase a home. There has been a general increase in homeownership over the period, mostly because this is preferred by households. FNMA suggests that its role was fundamental. Interfering events : A hurricane batters an area that is the target of a slum upgrading project and households and municipality must devote resources to recovery Hawthorne effect : Those in the treatment group respond to the treatment just because they receive it. Especially problematic where satisfaction is being measured—may only reflect that someone is paying attention to their problem.
Examples of each type of strategy Random assignment Housing Demand Experiment Component of the U.S. Housing Allowance Program -- objective to determine what share of a housing allowance subsidy would be spent on housing, i.e., improving their housing, and how much would be spent on other things. Note that the program required participants to live in in a unit meeting certain minimum physical standards--they could move to a unit meeting the standards.) -- Pittsburgh and Phoenix, selected for different market conditions: Pittsbugh was loose market and Phoenix a tight market. Random assignment to various sub-experiments Housing allowance vs. pure control (the control was given an equivalent cash payment not related in any way to the money being used for housing) Housing allowance vs. unconstrained cash payment Housing allowance programs with different parameters—both percent of rent and gap formulas .Matched constructed controls Schools defined to be similar--in terms of the test scores of pupils, income of families in the neighborhood, percent students from single parent households—are assigned to treatment and control groups. Students in the treatment group get intensive assistance with reading.
Statistically equated controls In project to test the impact of lower (subsidized) interest rates on the price paid for housing, samples are constructed of those receiving the subsidized loans and those paying market interest rates and having similar incomes. Analysis is of the price per sq.m. paid for housing after controlling for other borrower characteristics. Reflexive controls Measuring the impact of a training program by measuring training program participants’ work and pay experience before and after the training experience. (As we will see, this is probably not a reliable way to measure impact of a training program.) Reflexsive controls are especially at risk from the problem of “secular drift.”
The Jobs Corps is a long-established training program for young me and women. It is distinctive because participants are taken to rural locations--often retired military bases--where they receive intensive job training and counseling on appropriate life behavior and skills. The usual time at the training facility is six months. So the program is expensive. The source for this case is: Mathematica Policy Research, “Evaluation of the Economic Impact of the Jobs Corps Program: Third Follow-up Report.” Washington, DC: Office of Policy and Evaluation and Research, Employent and Training Administration, U.S. Department of Labor, 1982.
Pts. 1&2: These really are part of a process evaluation on how the program operated. This is a good example of the complementarity between implementation and impact evaluation.
The positive cost-effectiveness is based on all benefits, i.e., higher earnings and savings to society from lower crime rates (prosecution, court and prison costs).
Pt.2: Think of a regression model in which the observations include members of both the treatment group and the control group. This is a so-called “pooled regression.”
This is a program under which a young adult spends time consistently with young people who have been in trouble or are risk of being in trouble. The young adults are called mentors . This evaluation is described in
The basic question under analysis here is whether private firms selected through a competitive process would do a better job maintaining municipal housing that the city companies that had had a monopoly on providing these services for many years. The reference for this case study is: R. Struyk, K. Angelici and M. Tikhomirova,“Private Maintenance for Moscow’s Municipal Housing Stock: Does It Work?” Journal of Housing Economics , vol. 4, 1995, pp. 50‑70.
Two competitions were held--one in March 1993 and the second in September 1993. In March, the 2,000 units for which services were being procured were divided into 3 groups (packets) of about 700 units each. Three different firms won. In the September competition, 4 packets of units (about 1,250 each) were offered and 2 firms each won 2 packets.
For both the March and the September buildings, the first survey was conducted a few weeks before the private firm took over responsibility for maintaining the buildings. For the March buildings, two later surveys were conducted--one in May and one in November. For the September buildings, only one later survey was done, in January 1994. Importantly , interviewers returned to the same households in each survey round and almost always the same person in a household was interviewed. Thus, differences in perceptions and opinions among respondents should be minimized.
General points about the analysis The questions asked of respondents where to categorize a specific condition for a specified period of time. So one question was, “How often in the last month were the lights in your hallway not working?” They did not ask about satisfaction. X 2 test used throughout to test significance of the distributions from a null hypothesis of all answers being equally given. Maintenance, May-November. The mixed results were as follows: 6 of 11 categories got somewhat worse no change in 4 categories and improvement in 1 but some areas with deterioration in the scores are those where the firm itself was not fully responsible, e.g., secruity systems. Important note : During the period the city was very slow in paying the company and this may have led to reduction of effort. Recall that annual inflation in Russia at this time was about was 800%; so delays were very costly.
General points about the analysis The questions asked of respondents where to categorize a specific condition for a specified period of time. So one question was, “How often in the last month were the lights in your hallway not working?” They did not ask about satisfaction. X 2 test used throughout to test significance of the distributions from a null hypothesis of all answers being equally given. Maintenance, May-November. The mixed results were as follows: 6 of 11 categories got somewhat worse no change in 4 categories and improvement in 1 but some areas with deterioration in the scores are those where the firm itself was not fully responsible, e.g., secruity systems. Important note : During the period the city was very slow in paying the company and this may have led to reduction of effort. Recall that annual inflation in Russia at this time was about was 800%; so delays were very costly.
Private firms cost less . The terms of the competition was that the private firms had to bid a price less than the cost of services from the city firms. So by definition private firms cost less. Actually, initially private firms were about 10 percent cheaper. Over time in some cities, city firms cut their prices to be competitive. But in a number of cities the total savings in 30 percent or more from the former cost of city firms, adjusting for inflation.
Procedure is challenging because the analyst must assign dollar values to all costs and benefits. The simplicity of the B:C ratio is very attractive to decision makers because it provides a simple standard and the ability to compare projects explicitly.
Take a program with multiple benefits. For example, building a dam both protects a downstream city against floods and in this respect is credited with savings some lives, and it provides water for irrigation of farm land. So there are three benefits: urban property protected, lives saved, and additional food production. To get a unified estimate of the benefits the evaluator has to assign weights to the various benefits so that they can be added together--which is a highly subjective exercise. What weight to I assign saving a life versus five additional tons of corn production?
Lack of consistency Review of BCAs within the same agency found very wide differences in assumptions and measurement practices. Even greater among agencies. Results subject to manipulation This is a problem for projects for which B:C analysis is being done to assess whether the investment should be undertaken. One analysis of eight subway projects build recently in eight different U.S. cities found very systematic overstatements in the volume of passengers that would be carried by the new systems and large understatements of the costs of building and maintaining the systems. In the median case, capital and operating costs were underestimated by one-third and ridership was overestimated by 300%. So the average cost per rail passenger turned out to exceed the forecast in every case by at least 188% and in three cases by more that 700%. The structure of the Federal program that funds these projects contains strong incentives for such behavior: the agency ranks projects by their B:C ratios; often this is the most important criterion determining which projects are funded. Also, the Congress tends to allocate funds to specific ongoing construction projects to cover cost overruns.
Because of the uncertainty about the discount rate, governments often specify the rate to be used in assessing projects (which is based roughly on the kind of analysis described in brackets above). A reasonable value is the Government’s cost of long-term money--10+ year bonds. NPV = (B-C)1 + (B-C)2/(1+r) + (B-C)3/(1+r) 2 + …. [If use nominal interest rate, then do not need to deflate benefits and costs for inflation; opposite if one uses the real discount rate.] Note the sensitivity of the NPV in the table to the discount. This will always be the case when there is a large difference in the time profile for benefits and costs
