EVALUATION OF NATIONAL PROGARM

1. EVALUATION OF A NATIONAL
HEALTH PROGRAMME
PRESENTERS: Dr. Shanthosh Priyan
S
Dr. Joymati Oinam
MODERATOR: Dr. Shantibala K

2. OUTLINE OF PRESENTATION
1. Introduction
2. Monitoring & Evaluation definitions
3. Evaluation Vs Monitoring
4. History of evaluation
5. Why, What & When to evaluate
6. Types of evaluation
7. Principles of evaluation
8. The evaluation process
9. Conclusion

3. INTRODUCTION
Health services have become more complex
There has been a growing concern about their
functioning both in the developed and developing
nations
Questions are raised about the quality of medical care,
utilization and coverage of health services, benefits to
community health and improvement in the health status
of the recipients of care

4. An evaluation study addresses itself to these issues
So, program evaluation is an essential organizational
practice in public health
However, it is not practiced consistently across
program areas, nor is it sufficiently well-integrated into
the day-to-day management of most programs
Thus, a framework for understanding program
evaluation and facilitating integration of evaluation
throughout the public health system is needed

5. The purposes of this seminar is to
• summarize the essential elements of program evaluation;
• provide a framework for conducting effective program
evaluations;
• clarify the steps in program evaluation;
• review standards for effective program evaluation; and
• address misconceptions regarding the purposes and
methods of program evaluation

6. WHAT IS MONITORING?
Monitoring is the periodic oversight of the
implementation of an activity which seeks to establish the
extent to which input deliveries, work schedules, other
required actions and targeted outputs are proceeding
according to plan, so that timely action can be taken to
correct deficiencies detected
"Monitoring" is also useful for the systematic checking
on a condition or set of conditions, such as following the
situation of women and children

7. MEANING OF EVALUATION
Evaluation has its origin from the Latin word
“Valupure” which means the value of a particular
thing, idea or action
Evaluation, thus, helps us to understand the worth,
quality, significance amount, degree or condition of any
intervention desired to tackle a social problem
Evaluation means finding out the value of something
Evaluation simply refers to the procedures of fact
finding

8. WHAT IS EVALUATION?
“The systematic investigation of the worth, merit, or
significance of an ‘object’ ”
Michael Scriven
“The systematic collection of information about the
activities, characteristics and outcomes of programs to
make judgments about the program, improve program
effectiveness, and/or inform decisions about future
program development”
CDC

9. Evaluation has been defined as
“ The systematic and scientific process determining
the extent to which an action or set of actions were
successful in the achievement of pre-determined objectives.
It involves measurement of adequacy, effectiveness and
efficiency of health services”
-WHO

10. Relationship of Monitoring and
Evaluation

11. Monitoring
Recording
Analysis
Reporting
Corrective action at the operational
level
Information
Recording Recommendations
Analysis
Information
from other
sources
Information from
Monitoring
Affirmation or modification in
objectives, resources, and process
Relationship of Monitoring and
Evaluation
Evaluation

12. Difference between Monitoring
and Evaluation

13. Objective
Monitoring Evaluation
To track changes from
baseline conditions to
desired outcomes
To validate what results were
achieved, and how and why
they were or were not
achieved
Methodology Tracks and assesses
performance through
analysis and
comparison of
indicators over time
• Evaluates achievement or
outcomes by comparing
indicators before and after
the intervention
• Involves Value Judgment
• Relies on monitoring data
and information from
external sources

14. Characteristics
Continuous and
systematic by
Programme/ Project
managers and key
partners
Time-bound, periodic, in-
depth by Internal or
External evaluators and
partners
Uses Alerts managers about
problems in
performance, provides
options for corrective
actions and helps
demonstrate
accountability
Provides managers /
donors / stakeholders with
strategy and policy
options, provides basis for
learning and demonstrates
accountability

15. PROGRAM
“An organized response to eradicate or eliminate or
reduce 1/more problems where the response includes 1/
more objectives & the expenditure of resources”
“Any set of organized activities supported by a set of
resources to achieve a specific and intended result.”

16. Research vs. Evaluation
Knowledge intended for use
Program- or funder-derived
questions
Judgmental quality
Action setting
Role conflicts
Often not published
Multiple allegiances
Production of generalizable
knowledge
Researcher-derived questions
Paradigm stance
More controlled setting
Clearer role
Published
Clearer allegiance
Research Evaluation
Systematic
Methods

17. “Research seeks to prove,
evaluation seeks to improve…”
M.Q. Patton

18. HISTORY OF EVALUATION
Began in the field of education
Strengthened during the 1960’s emphasis on social
programs and determining their effect on society
Further strengthened during the 1990’s emphasis on
outcomes measurement and quality improvement

19. WHY EVALUATE PROGRAMS?
To gain insight about a program and its operations – to see
where we are going and where we are coming from, and to
find out what works and what doesn’t
To improve practice – to modify or adapt practice to enhance
the success of activities
To assess effects – to see how well we are meeting objectives
and goals, how the program benefits the community, and to
provide evidence of effectiveness
To build capacity - increase funding, enhance skills,
strengthen accountability

20. WHAT CAN BE EVALUATED?
Direct service
interventions
Community mobilization
efforts
Research initiatives
Surveillance systems
Policy development
activities
Outbreak investigations
Laboratory diagnostics
Communication campaigns
Infrastructure-building
projects
Training and educational
services
Advocacy works
– MMWR, 1999
Framework for Program Evaluation in Public
Health

21. WHEN TO CONDUCT EVALUATION?
Conception Completion
Planning a
NEW
program
Assessing a
DEVELOPING
program
Assessing a
STABLE,MATURE
program
Assessing a
program after
it has ENDED

23. TYPES OF EVALUATION

24. • Formative evaluation
 Evaluation intended to improve performance, most
often conducted during the design and/or
implementation phases of projects or programs
 Its intent is to assess ongoing project activities
 Formative evaluation has two components:
implementation evaluation and progress evaluation
Implementation(Process) evaluation:
 The purpose of implementation evaluation is to assess
whether the project is being conducted as planned

25. Progress evaluation:
 To assess progress in meeting the goals of the
program and the project. It involves collecting
information to learn whether or not the benchmarks of
participant progress were met and to point out
unexpected developments

26. Evaluation Area
(Formative assessment )
Evaluation Question Examples of Specific
Measurable Indicators
Staff Supply Is staff supply sufficient? Staff-to-client ratios
Service Utilization What are the program’s usage
levels?
Percentage of utilization
Accessibility of Services How do members of the target
population perceive service
availability?
• Percentage of target
population who are
aware of the program in their
area
• Percentage of the “aware”
target population who know
how to access the service
Client Satisfaction How satisfied are clients? Percentage of clients who
report being
satisfied with the service
received

27. • Summative(Outcome/Impact) evaluation
An evaluation conducted at the end of an intervention
to determine the extent to which anticipated outcomes were
produced such as improved survival or reduced disability
The traditional outcome components are the 5 D’s of ill
health, viz. disease, discomfort, dissatisfaction,
disability and death

28. Evaluation Area
(Summative Assessment)
Evaluation question Examples of specific
measurable indicators
Changes in Behaviour Have risk factors for
cardiac disease have
changed?
Compare proportion of
respondents who reported
increased physical activity
Morbidity/Mortality • Has lung cancer
mortality decreased by
10%?
• Has there been a
reduction in the rate of
low birth weight babies?
• Age-standardized lung
cancer mortality rates for
males and females
•Compare annual rates of
low-birth weight babies over
five years period

29. • Participatory evaluation
Evaluation which provides the active involvement of
those with a stake in the program, (providers, partners,
beneficiaries, and any other interested parties) in designing,
carrying out and interpreting an evaluation
• 360 degree Evaluation (evaluation by planner &
implementers)
An evaluation by those who are entrusted with the
design and delivery of a project
TYPES OF EVALUATION Contd..

30. PURPOSE OF EVALUATION
To improve health programs and the services for
delivering them
To guide the allocation of resources in current and
future programs
Should be used constructively & not for the
justification of past actions/merely to identify their
inadequacies

31. Decision-oriented tool and to link closely with
decision making, whether at the operational/the
policy level
The very process of carrying out an evaluation often
induces a better understanding of the activities being
evaluated, and a more constructive approach to their
implementation and to any further action required

32. CONSTRAINTS OF EVALUATION
Evaluation, difficult in any field, presents particular
problems in health work owing to the very nature of the
activities, which often do not lend themselves easily to
the measurement of what has been attained against
predetermined, quantified objectives
It is therefore often unavoidable to apply qualitative
judgment, supported by reliable, quantified information

33. Changes in a health situation are often brought about by
elements outside the health sector, making evaluation,
particularly of effectiveness & impact, even more
difficult
Development and selection of sensitive and specific
indicators a challenging and difficult task
There is a certain in-built resistance in principle to
accepting evaluation & its results as a valid management
tool

34. PLACE OF EVALUATION IN THE
HEALTH DEVELOPMENT PROCESS
The purpose of the managerial process for national
health development is to build up the health system in a
rational & systematic way
Health program evaluation is part of the managerial
process for national health development

36. RESPONSIBILITY OF EVALUATION
Central level: Director-general of health services in the
ministry of health, the minister of finance, Directors, Deans
District level: Directors of district hospitals, public health
laboratories, environmental health services, training schools
& finally, the district health officer
Local level: Communities themselves

37. COMPONENTS OF EVALUATION
PROCESS
Relevance:
Relates to the appropriateness of the services, whether it
is needed at all
For example, vaccination against small pox is now
irrelevant because the disease no longer exists

38. Adequacy:
Implies that sufficient attention has been paid to certain
previously determined courses of action, such as the
various issues to be considered during broad
programming
For example, the staff allocated to a certain program
may be described as inadequate if sufficient attention
was not paid to the quantum of workload and targets to
be achieved

39. Accessibility:
Proportion of the given population that can be expected
to use a specified facility, service, etc.
The barriers of accessibility may be physical(distance,
travel, time); economic(travel cost, fee charged); or
social and cultural(caste or language)
Acceptability:
The services provided may be accessible but not
acceptable(male sterilisation, rectal cancer screening)

40. Efficiency:
Expression of the relationships between the results
obtained from a health program/activity and the efforts
expended in terms of human, financial and other
resources, health processes and technology, and time
(number of immunizations provided in an year as
compared with an accepted norm)
Cost-benefit analysis will be useful for this purpose
To find out if optimal utilization of available resources is
being made

41. Effectiveness:
Expression of the desired effect of a program,
service, institution or support activity in reducing a
health problem or improving an unsatisfactory health
situation
Measures the degree of attainment of the
predetermined objectives & targets of the program,
service/institution

42. Impact:
Expression of the overall effect of a program,
service/institution on health and related socio-
economic development
Aimed at identifying any necessary change in the
direction of health programs so as to increase their
contribution to overall health and socioeconomic
development

43. Sustainability:
Meeting needs without compromising the ability of
future generations to meet their needs (project will
continue after donors intervention)

44. FREQUENCY OF EVALUATION
While evaluation is a continuing process, its results have to
be summarized & reported on at given times/specified
intervals
The frequency will vary ranging from relatively short
intervals for the assessment of progress and efficiency to
much longer intervals for the assessment of effectiveness
and impact

45. INDICATORS AND CRITERIA FOR
EVALUATION
Indicators are variables that help to measure changes
They are evaluation tools which can measure changes
directly/indirectly
Indicators should be:
 valid
 reliable
 sensitive &
 specific

46. Health policy indicators:
Allocation of adequate resources to health for all
Level of community involvement in attaining health for
all
Social and economic indicators:
Rate of population increase
Adult literacy rate
Adequacy of housing as expressed as number of persons
per room

47. Indicators of provision of health care
Availability
Physical accessibility
Utilization of services
Health status indicators
Infant mortality rate
Child mortality rate
Under 5 mortality rate
Life expectancy

48. Criteria are standards by which actions are measured
2 types- Technical and Social
Technical-normally highly specific to program
For example, technical criteria for the guarantee of
safety of drinking water would be certain technical
standard for purity of drinking water

49. A social criteria for the guarantee of the continuation of
the water supply would be the existence of community
organizations for the maintenance of the supply
Main purpose of criteria is to provoke thought leading
to judgment
Indicators and criteria should be included in the
program at the planning stage, so that the information
requirements can be determined early on

50. INFORMATION SUPPORT
Evaluation has to be based on valid, relevant and
sensitive information
The types of information required may include
political, social, cultural, economic, environmental
and administrative factors influencing the health
situation as well as mortality and morbidity statistics

51. STANDARDS FOR EFFECTIVE
EVALUATION
Utility: Who needs the information and what information
do they need?
Feasibility: How much money, time, and effort can we
put into this?
Propriety: What steps need to be taken for the
evaluation to be ethical?
Accuracy: What design will lead to accurate
information?

52. Utility:
Ensures that the information needs of intended users are
met.
Who needs the evaluation findings?
What do the users of the evaluation need?
Will the evaluation provide relevant (useful) information
in a timely manner?

53. Feasibility
Ensures that evaluation is realistic, prudent,
diplomatic, and frugal.
Are the planned evaluation activities realistic given
the time, resources, and expertise at hand?

54. Propriety
Ensures the evaluation is conducted legally, ethically,
and with due regard for the welfare of those involved
and those affected.
Does the evaluation protect the rights of individuals
and protect the welfare of those involved?
Does it engage those most directly affected by the
program and by changes in the program, such as
participants or the surrounding community?

55. Accuracy:
• Ensures that the evaluation reveals and conveys
technically accurate information.
• Will the evaluation produce findings that are valid and
reliable, given the needs of those who will use the
results?

56. THE EVALUATION PROCESS

57.  The evaluation process includes a number of
distinct stages-
A. planning evaluations
B. gathering or recording baseline data
C. managing evaluations
D. implementing evaluations
E. using evaluations

58. A.Planning Evaluations
 The evaluation plan should identify:
Why
When
What
Who
How
Resources

59. Why do the evaluation?
Identify number of evaluation purposes
Identify who needs what information and for what
purpose
 Four common reasons for evaluations are:
Formative Evaluation
Summative Evaluation
Learning Lessons For Future Application
Accountability

60. When to do an evaluation?
 The timing of a major evaluation is affected by:
its relation to the sector/ministry plan, within the
country development plan
a significant problem identified in the course of
monitoring
a donor request

61. Evaluation Matrix

62. What should be evaluated?
 The scope of work of those conducting
evaluations-
Description: What happened and how does this
compare with what was expected?
Analysis: Why and how did it happen or not
happen?
Prescription: What should be done about it?

63.  An evaluation may focus on different levels of
results of a service/programme or project-
Inputs-outputs
Processes
Outcomes Or Impacts

64. Development of a frame logical model
It is a flow chart that shows the program’s components, the
relationships between components and the sequencing of
events.

66.  The following key issues provide another focus in
formulating the main questions an evaluation
should address:
Effectiveness - Is the project or programme
achieving satisfactory progress toward its stated
objectives?
Efficiency - Are the effects being achieved at an
acceptable cost, compared with alternative
approaches to accomplishing the same objectives?
Relevance - Are the project objectives still
relevant?

67. Impact - What are the results of the project?
Sustainability - Is the activity likely to continue
after
donor funding, or after a special effort, such as a
campaign, ends?

68. Sample evaluation questions: What
might stakeholders want to know?
Program clients:
• Does this program provide us with
high quality service?
• Are some clients provided with
better services than other clients? If
so, why?
Program Staff:
• Does this program provide our
clients with high quality service?
• Should staff make any changes in
how they perform their work, as
individuals and as a team, to
improve program processes and
outcomes?
Program managers:
• Does this program provide our
clients with high quality service?
• Are there ways managers can
improve or change their activities, to
improve program processes and
outcomes?
Funding bodies:
• Does this program provide its clients
with high quality service?
• Is the program cost-effective?
• Should we make changes in how
we fund this program or in the level
of funding to the program?

69. Who will do the evaluation?
 The evaluation plan should:
identify which entity (entities) will
manage/supervise the evaluation
indicate what type of person (s) should conduct it,
i.e., internal or external evaluators or a combination

70. How to answer the evaluation
questions?
 Determining information needs is an initial
step-
 Existing data should be identified -
Monitoring documents, previous evaluations, and
other documentation (audits, mid-term/annual
reviews) of the project
Government routine reporting systems records or
evaluations of similar programmes in agency or
donor offices (WHO, other ministries, NGOs, etc.)

71. Identify the minimum amount of new information
needed to answer the evaluation questions
Effective evaluations concentrate on collecting
timely, relevant and useable data

72.  Decide what data should be used for the
evaluation-
While planning an evaluation, the sponsors should
work to specify evaluable questions and design a
feasible data collection plan
Involves deciding which indicators to use to
measure
progress

73.  Decide on criteria to judge progress-
Every evaluation should measure progress and
compare it to some standard-
 programme/project objectives
 past performance
 national targets
 baseline data
 similar services or project areas

74.  Match data-collection methods with evaluation
purposes
There are four fundamental ways of obtaining
information:
1. collecting, tabulating, reviewing already available data
2. questioning people through interviews, focus groups
3. conducting surveys
4. observing people and things though field visits

75. Types of Data, Uses, and Collection Methods
for different Evaluation Focuses
Focus Type of Data Use/
purpose
Data Collection
Methods
Input Financial, material,
personnel
Delivered to
project?
Administrative
records
Output Services provided
and used
Reach target
group?
Administrative
records, RAP,
surveys
Outcome
/impact
Change in
beneficiary status
Effects
attributable
to
programme?
Routine
reports,RAP,
survey, views of
informed people
Efficienc
y
Costs of input,
outputs, impacts
Most effect
for cost?
Cost-
effectiveness
comparisions

76. What resources are needed and
available for evaluation?
Early in planning an evaluation make an estimate
of its costs
Costs depend on the nature and size of the service
/programme and design of the evaluation
Evaluation planners should consider the non-
financial or
indirect costs of the evaluation
The amount of resources available, determined
during programme planning, should be built into the
budgets of all the entities supporting the

77. B.Data sources for
Evaluation Existing data includes-
Primary and secondary data
Data from the service or project -
monitoring documents [progress reports and field trip
notes] & evaluations [midterm reviews]
Relevant outside information include
evaluations of similar programmes,
special studies and researches of the problem
data from government census, surveys, sentinel

78. C.Managing Evaluations
 Principally consists of -
negotiating the evaluation plan
preparing the Terms of Reference
selecting and working with the evaluation
team

79. Negotiating the evaluation plan with
others
Negotiating skills and understanding how to
design evaluation
The various roles must be taken into account in
deciding who should be consulted in determining
evaluation objectives and methods

80. Drawing Up the Terms of Reference
The Terms of Reference, sometimes called a
Scope of Work (SOW), set out the formal
agreements about the evaluation
It is like a contract with the evaluators spelling out
what they are to accomplish
Prepared four to six months before the evaluation
to allow adequate time for planning
A good TOR paves the way for a good evaluation

81. Suggested Contents of the Terms of Reference
1. Background and Purpose
2. Evaluation Questions or Objectives
3. Evaluation Methods
4. Composition of the Evaluation Team
5. Schedule of Major Tasks
6. Deliverables
7. Financial Requirements and Logistical Support

82. Recruiting and selecting the
evaluation team
Include people from the country on the team
If women are the subject of the study, recruiting women
team members
Size of the team depends also on the duration of the
evaluation
Steering committee members can help recruit
Finally, discuss TOR with those selected and make any
mutually agreed upon modifications before initiating
contractual arrangements

83.  Working with the evaluation team
 The person(s) designated to manage the
evaluation and supervise the evaluation team
should meet with the team before it begins-
to give instructions
review the work plan and answer questions
to send background materials to team members
before the orientation meeting

84.  Supervising the evaluation team
Regular contact with the team leader
Maintaining frequent communication and a
cooperative relationship enables problems to be
identified early, tackled together, and the work
plan
modified accordingly

85. Finally, supervisors should evaluate the work of
the evaluation team and assess the quality of the
evaluation and report by:
a. providing feedback to team members, individually
or in a group.
b. asking team members to offer their suggestions
for
improving the process next time
c. assessing the evaluation report and discussing its
strengths and limitations with the team

86. D.Conducting
Evaluations
Tasks of Evaluators
(1) familiarize themselves and refine the evaluation plan
(2) gather data
(3) analyse existing data and those collected to formulate
findings and recommendations
(4) write the evaluation report
(5) debrief the interested parties on the findings and
recommendations
(6) Disseminate evaluation report

87.  Refining the evaluation plan
Evaluators must choose the type of sample, the
sites and sample size
Site selection is required for most evaluations, and
should be done carefully so that sites are
representative

88.  Formulating Findings
Findings may be called conclusions or lessons learned-
a. describe project/programme/service results
b. compare them to what was planned and/or some
other
standard
c. judge whether "enough progress" was made
d. identify major reasons for successes, failures and
constraints

89.  Developing recommendations
Start with the findings
Directed to different kinds of decision makers
Avoid vague, general and impractical
Always list in priority order and include costs of
implementing them
A proposed timetable

90.  Preparing the Evaluation Report
Suggested Contents of Evaluation Report -
1. Title page
2. Table of Contents
3. Acknowledgments (optional)
4. Executive Summary
5. Introduction
6. Evaluation Objectives and Methodology

91. 7. Findings
8. Recommendations
9. Lessons Learned (optional)
10. Appendices

92. E.Using Evaluation
Results
Develop a new/revised implementation plan in
partnership with stakeholders
Monitor the implementation of evaluation
recommendations and report regularly on the
implementation progress.
Plan the next evaluation

93. Conclusions
Monitoring and evaluating are key functions to
improve the performance of those responsible for
implementing health services
During programme preparation and start up it is
necessary to specify general arrangements and
make budgetary provisions for future evaluations

94. The evaluation plans should be outlined in the
country
programme plan of operations, annual plans of
action and
annual reports for each programme
Interventions that are effective in developed
countries may
not be effective in developing countries
Rigorous program evaluation of interventions in
various
resource-limited settings is needed to determine
which