Data sharing and data-driven decision making are a critical component for successful collaborations that drive toward student achievement. At this session, we will discuss best practices for developing a data driven, results-based organization, learning from Higher Achievement’s experiences successfully submitting to a third party evaluation, customizing a management information system for in-house use, and regularly using internal and external data to make strategic and programmatic decisions.

1. Evaluation from Top to Bottom
Rachel Gwaltney
Chief of Programs, Higher Achievement
www.higherachievement.org

2. AGENDA
• Program overview
• Third-party research study findings
• Internal data analysis
– Participant outcomes
– Program quality
– Staff
• Interactive discussion
– Benefits and best practices for data and systems
– Challenges of data and systems
• Resources

3. Higher Achievement’s Theory of Change
• After school and summer program
Increased
offering middle school students
Academic
650 hour extended learning
Opportunities
beyond 900 hours of school
Increased • Preparing scholars for college and
Increased
Academic Academic career readiness
Achievement Interest • Combined culture and content
model
Increased • Founded 35 years ago in DC
Academic • Started national expansion 2008
Effort

4. Who Are Our Scholars? Scholars commit to
• 5th – 8th grade 650 hours per
• Starting GPA: 2.5 year, beyond the
1000 hours in
• 99% minority
school
• 81% FARM-eligible
• 79% will be
first-generation college
graduates
• Most are recommended
by teachers

5. Culture
- high expectations
- praise for effort
- student voice and
choice
- learning is fun!
http://www.youtube.com/watc
h?v=EOEZZvI2aKU&feature=rel
ated

6. Results and Impact
Annual outcomes: Third-party research:
• Significant improvements in • The intensive year-round
grades, test scores, and school program had a significant
attendance impact on youth's standardized
reading and math test scores.
8th grade graduates (2010):
• Improved their average GPA • 64%of parents of children
from 2.2 to 3.2 attending the program
• 95% were placed in a top high confirming at their first-year
school program follow-up that they spoke to
Higher Achievement staff
• 85% improved or maintained about their child's progress at
an A or B in math and reading least once a month.

7. Research Partners
• Principal Investigators
– Carla Herrera (Public/Private Ventures)
– Jean Baldwin Grossman (P/PV, Princeton)
– Leigh L. Linden (The University of Texas at Austin)
• Funders of published work to date
– The Atlantic Philanthropies
– The William T. Grant Foundation
– The Wallace Foundation
• Data Collection
– Survey Research Management

8. Research design
• Overview
– Evaluation of the Higher Achievement program
• After-school and summer program
– One-year, two-year and summer findings
– Four-year evaluation in progress
• Recruitment and Randomization
– 951 students applied to Higher Achievement
– More students applied than Higher Achievement could serve
– Randomly chose students to offer admission to Higher Achievement
– Remainder became a control group
• Advantages of design
– Gold standard evaluation strategy
– Sample comprises “types” of children served by Higher Achievement

9. Outcomes Measured
• Key outcomes and variables of interest:
– Standardized test scores
• Abbreviated SAT 10 Problem Solving
• Abbreviated SAT 10 Reading Comprehension
– Behavior
– Academic attitudes
– Perceptions of peer and adult support
– Participation in Higher Achievement and other OST
programs
– Activities related to high school application
• Analyzed separately:
– Parent and child assessments of OST programs
– Mentor and teacher surveys within Higher Achievement
– Qualitative data on Higher Achievement

10. Timing of Data Collection
2006 2007 2008 2009 2010 2010 2011 2012
Spring Spring Spring Spring Spring Fall Spring Spring
Cohort 1 (N=276)
Survey Round Baseline FU1 FU2 FU4
Grade Entering 5th/6th 6th/7th 7th/8th 9th/10th
Cohort 2 (N=276)
Survey Round Baseline FU1 FU2 FUSp FUFa FU4
Grade Entering 5th/6th 6th/7th 7th/8th 8th 8th 9th/10th
Cohort 3 (N=399)
Survey Round Baseline FU1 FU2/FUSp FUFa FU4
Grade Entering 5th/6th 6th/7th 7th/8th 7th/8th 9th/10th
Note: FU1 = One-Year Follow-Up
FU2 = Two-Year Follow-Up
FU4 = Four-Year Follow-Up
FUSp = Spring FU for the Summer Study
FUFa = Fall FU for the Summer Study

11. Standardized Test Scores
• Significant effects after two years
– Problem Solving: 0.12 Standard Deviations
– Reading Comp: 0.09 Standard Deviations
• Effect sizes are larger than those reported for other
OST programs evaluated by large-scale RCT studies.
• No effects after one year
• No difference during summer 2010

12. Behavior
• Asked youth about their engagement in several negative
behaviors
– In-school: e.g., principal’s office, tardies, skipping
– Out-of-school: e.g., taking or breaking something, hitting
• At both the one- and two-year follow-ups treatment
students were more likely to report engaging in some of
these behaviors.

13. Academic Attitudes
• Six measures
– Industry and Persistence
– Creativity
– Self-Perceptions of Academic Abilities
– Enjoyment of Learning
– Curiosity
– Ability to Change the Future through Effort
• Overall, treatment students have more negative attitudes than
control students after the first year.
• No overall differences at the second year.
• Effects vary by the grade at which youth enter HA.
• Gains in Enjoyment of Learning during Summer 2010

14. Program Participation
• Higher Achievement provides opportunities that
scholars would not otherwise have.
– Without access to HA, 35 percent attend an academic OST;
– Access to HA increases this by 52 percentage points.
– Treatment students average more time in academic OST
programs
• 10.3 hours more a week during the academic year
• 19.8 hours more a week during the summer

15. Activity Participation
• Treatment youth were more likely to report engaging in
a wide range of activities. For example:
– Visiting a college campus
– Speaking to a group about youth’s ideas or work
– Speaking to an adult about high school, college and
jobs
– Going to events outside youth’s neighborhood
– Writing poems, stories, etc. not for school
– Going to events outside of school

16. High School Application Activities
• Only tested in Summer 2010 and four-year follow-up
• Students were more likely to report engaging in various
preparatory activities. For example:
– 14 percentage point difference in visiting high schools
– 15 percentage point difference in getting application information
on a school
• Significant increase in students wanting to attend
competitive high schools.
– A relative increase of 16 percentage points

17. How are these outcomes achieved?
• This study cannot rigorously answer this question.
• But Higher Achievement has several characteristics that
make it stand out as a strong program:
– Long-term and intensive
– Broad range of academic and enrichment activities
– Guided by grade-level curricular standards
– Staff are well trained and supported
– Strives to involve parents
– Focus on small-group instruction
– Opportunities for leadership

18. Conclusions
• Participation in well-structured, long-term, academically focused
out-of-school-time programs can boost student achievement.
• Gains take time, emerging only after two years of access to the
program.
• Gains coincided with increased reports of negative behavior and
without an improvement in academic attitudes.
– Requires further investigation
• Engagement in activities related to high school application process is
promising.
– Fourth-year data collection to be completed this summer and
published the following year.
• Lack of test score differences in summer does not mean that the
summer program is not an important component of the program.

19. Implications:
• Organizations
• Sector
• Policy

20. Lessons Learned
• Addressing moral question of denying access to
program for research purposes
• Ensuring staff capacity to recruit and support
• Feedback from researchers was invaluable for
program improvement
• Retention is critical
• Plan regarding communicating findings
• Work with research team to secure investment

21. Internal Data Practices – Scholar Data
FREQUENCY DATA TRACKED
Daily Individual feedback on session
participation and progress
Quarterly Report card data (grades, attendance)
Scholar Action Plan
Biannually (twice/year) Attitudes and behavior (360 survey)
Annually Scholar outcomes: Standardized test
scores, GPA, school attendance, high
school placement

22. Internal Data Analysis - Program
FREQUENCY DATA TRACKED
Weekly Feedback from volunteer mentors
Biweekly (every two weeks) Dashboard data
Periodically Program quality observations
• Internal tool correlated to core program
elements
• YPQA external tool
Triannually (three times/year) Quality assurance reports compiled from
observations
Annually Scholar outcomes

23. Internal Data Analysis - Staff
FREQUENCY DATA TRACKED
First 90 days Completion of orientation goals
Quarterly Progress toward workplan goals
Annually • Evaluation against workplan goals and
organizational culture
• Scholar outcomes

24. Lessons Learned
• Invest in the right systems
• Set up and enforce strong systems for
data collection
• Train staff to report on, understand, and
act on analysis of data
• Make data-driven improvement part of
organizational culture

25. Discussion

26. Resources
• Harvard Family Research Project: www.hfrp.org
– Afterschool Evaluation 101
• ChildTrends: www.childtrends.org
– Data-Driven Decision-Making in Out-of-School Time
Programs
• Forum for Youth Investment: www.forumfyi.org
– From Soft Skills to Hard Data
• Wallace Foundation: www.wallacefoundation.org
– Hours of Opportunity
– The Cost of Quality OST Programs
• Public/Private Ventures: www.ppv.org

27. Thank You!