2010 NSF User Friendly Handbook for Project Evaluation, Chapter 4

1. The 2010 User Friendly
Handbook of Project
Evaluation
Chapter 4: The Evaluation
Process: Carrying Out the
Study and Reporting

2. Presented by
M. Bess-Frazier
T. Flemming
V.Lee
A. Smith
N. Wooten

3. Introduction
The Process of Evaluation and Reporting
consists of 4 general steps:
• 1. Data Collection
• 2. Data Analysis
• 3. Report the Findings
• 4. Publishing the information

4. Step 1: Data Collection

5. Step 2: Analyze the Data

6. Step 3: Report the Findings

7. Step 4: Publish (Disseminate) the Information

8. Step 1: Data Collection

9. Conducting the Data Collection
Things to consider:
• Get permission
• Consider needs and sensitivities of
participants
• Make sure data collectors are properly
trained, unbiased, and objective
• Get data from as large of a sample as
possible
• Disrupt as little as possible
( Frechtling, 2010, p. 39)

10. The Necessary Clearances and
Permissions
Think about the setting of your data
collection!
• Talk to teachers, administrators, bosses,
or parents where applicable
• Consider permission forms
• Consider who needs a copy of the final
data collection and conclusion
It may help with cooperation if you offer to
share the results after you’re done!

11. Consider the Needs and Sensitivities of
Participants
• Be honest with participants
• Assure participants that no personal
repercussions will result from any
information provided
As an anecdote, consider a survey on what
percentage of the population uses their seatbelt
when in a care. Do you think the results would
differ if a police officer gave the survey?

12. Make Sure Data Collectors are
Properly Trained
• Provide data collectors information about the
community or culture where they will be
collecting
• Data collectors need to ask the same
questions and use the same prompts
• They should be supervised to avoid
“coaching” and a distortion of data
• Data collectors should be unbiased

13. Collect Data from as Many
Members as Possible
• More members will decrease the margin
of error (Moore, 1993)
• A response rate of 70% or higher within
the sample is considered to be high
quality
• Follow up on non-participants

14. Avoid Disruption
Consider the setting of your data collection!
Consider the schedules of participants and
the project as well.

15. Step 2: Analyze
the Data

16. Data analysis checklist
✓Check the raw data and prepare them for
analysis
✓Conduct initial analysis based on the
evaluation plan.
✓Conduct additional analyses based on
the initial results.
✓Integrate and synthesize findings.

17. Check data for unusual responses.
– Example: selecting
more than one answer
instead of the one
required
– Choosing the same
response for all
question (i.e. “c”)
– Inconsistent answers
These questions may need to
be eliminated from the data that
will be analyzed. The data is
then prepared for coding and
entering for computer analysis.
Verification and quality control
procedures should already be in
place.
✓Check the raw
data and prepare
them for analysis.

18. – The analysis may raise
more questions as results
are calculated. Be careful
to stay with the set of
analysis that were
originally of interest.
– Updated software is better.
These offer support to
analysts and a way to
manage large sets of
narrative data.
– quantitative data collection
should be consistent to
avoid unnecessary
invalidation or affect the
evaluation integrity.
✓Conduct initial
analysis based on
the evaluation plan

19. “It is very likely that the initial analyses
will raise as many questions as they
answer.”
– The second analyses will address
these.
» Example: “The first analysis
looked at teacher performance, a
second analysis might subdivide
the total group into subunits of
particular interest—e.g. more
experience versus less
experienced…..examine if any
significant differences were found
between them. “
» These cycles can continue as
resulting data suggest other
interesting avenue to explore;
even if they were not planned
(Frechtling, 42).”
– Available time and money determines
which tasks can be completed.
✓ Conduct
additional
analysis based
on the initial
results.

20. Integration of the separate analyses,
the development of the findings’
conclusions are the last steps of
data analysis. The results may not
produce completely consistent
findings or fit into a report that
explains ambiguities. Some report
findings can remain unanswered...
✓ Integrate
and
synthesize
findings

21. Step 3: Report the Findings
● Requires:
o pulling together the collected data
o condensing the findings to correlate the
original questions of the evaluation
o publicize the findings

22. Main Sections of Formal Reports
“1. Background
2. Evaluation study questions
3. Evaluation procedures
4. Data analyses
5. Findings
6. Conclusions (and recommendations) “
(Frechtling,44)

23. Background
This section can include:
• a need or focus of the project and objectives
• possibly a journal or literature review
• the stakeholders and their background
• the study participants
• the activities and components
• project expected timeframe and venue
• the project resources used
• the expected measurable outcomes

24. Evaluation Study Questions
• Lists the questions that the study
addressed.
• Also includes variables that may affect the
project outcome, for instance, issues with
data collection, time constraints.

25. Evaluation
Procedures ● This section also describes the
types of data collected and the
instruments used for the data
collection activities.
● It is helpful at the end of this
section to include a matrix or
table that summarizes the
evaluation questions, the
variables, the data collection
approaches, the respondents,
and the data collection schedule.
● This section
describes the
project
participants.
● The participants’
selection method
and whether
sampling was
used.

26. Data Analyses
Section Explains:
• Data analyses methods
• Data analyses stages and how they were applied
• Data integrity safeguards to exclude study confounds
• Description of measures to ensure the study
participants represented the subject population
• A table summarizing the data analyses, if feasible

27. Findings
• Includes the analyses results
• Lists the study/ project questions asked even if no
answer was produced from the study
• Visual displays such as graphs if applicable
• A final project summary that explains the major
conclusions

28. Conclusions and Recommendations
• Summarizes more generally
• Addresses the project’s findings,
evaluation questions and program goals

29. Other Sections
• If applicable , there can also be formal
reports that have:
– A study abstract with findings summary
– An executive summary which contains
more detail as to the study synopsis,
conclusions, including implications for
relevant practices

30. Developing an Evaluation Report
• Ask colleagues or stakeholders for a critical
review of the report and any suggestions for
improvement prior to the final publication
• The report should be presented in a way
that is suitable for intended audience

31. Step 4: Disseminating the Information

32. ● Dissemination is the last part of project
evaluation. (Spreading your information)
● It should include the project funding source or prospective funding sources

33. The Dissemination should have:
• What the group needs to know and the best
way to communicate the information to them
• In what way did the project accomplish the
goals according to the National Science
Foundation?

34. Sharing your Findings
• Make a list of the various audiences to
share findings.
• It is important to share projects that
worked and projects not so fruitful.

35. Conclusion
An effective study evaluation should include
collecting data, analyzing the data, and reporting and
publishing the study information. There are various
elements to consider to maintain integrity and avoid
biased in collecting study data. Objectivity in reporting
the study evaluation findings to stakeholders, school
district administrators and study funding sources will
enhance the ongoing development of educational
research for all professionals.

36. References
Frechtling, J., Mark, M., Rog, D., Frierson, H., Thomas,
V., Hood, S., Hughes, G. (2010). The 2010 User Friendly
Handbook of Project Evaluation. Arlington, VA: National
Science Foundation, Directorate for Education & Human
Resources, Division of Research, Evaluation and
Communication.
Moore, D., & McCabe, G. (1993). Introduction to the
practice of statistics (2nd ed.). New York: Freeman.

37. Open Ended Questions for the
Class
● In data collection, we must consider the needs and
sensitivities of the participants. For example, it would
be biased to have a police officer ask people if they
broke the law. What other examples can you think of
where the data collector might affect the information
received? How might this affect us in the classroom
as teachers?
● For Reporting the Findings and Study Dissemination,
what are the most important factors to consider in how
and to whom the findings are reported?