An overview of the MURSD 1:1 Learning program evaluation.

1. Getting Started
• Get an index card & move to the poster with the writing that matches
the color of your index card
• Introduce yourself to the other participants at the poster
• Think about three core values or skills that shape your thinking about
1:1 computing in your classroom/school/district
– MURSD’s 1:1 Learning Core Values: Communicate, Collaborate, Innovate
• Record your three values on the chart paper and explain your
reasoning to your peers

2. Evaluating Your 1:1 Learning Program: One District’s
Journey
Dr. Joseph Maruszczak
Mendon - Upton Regional School District
David Quinn
Mendon - Upton Regional School District
Dr. Damian Bebell
Boston College

3. Goals for Today’s Session
1. Provide participants with an structure for
conducting an evaluation of 1-1 Learning
in their classroom, school or district.
2. Participants will identify core values that
drive their beliefs about 1:1 Learning in
their respective context.
3. Participants will begin to design questions
around their core values that will help
them to determine their current successes
and potential next steps.
4. Participants will connect with other
teacher-leaders and administrators to
share questions and ideas for conducting
an evaluation

4. Historical Context: The Why
The Mendon-Upton Regional School District uses technology to communicate, collaborate,
and innovate, creating an exceptional, personalized learning experience so all will realize
their potential.

5. Historical Context: The
Timeline
- Fall 2011: Pilot launched for Grade 7 students
(90 total)
- Fall 2012 & 2013: 1:1 Learning for all students
in Grades 6 – 8
- Fall 2014: 1:1 Learning for all students in
Grades 5– 12
- Fall 2016: Proposals solicited for external 1:1
Learning program evaluation
- Fall 2017: Program evaluation released

6. Evaluation Process: What do We Want to
Know?

7. Communicate:
1. How are iPads being used by teachers and students to communicate with others?
2. How do our MURSD practices compare to other districts with 1:1 programs?
3. How might we capture and measure our teachers’ and our students’ growth in their ability to
communicate using devices?
4.What next steps are recommended to utilize the devices to enhance communication for
student learning?
Collaborate:
1. How are iPads being used by teachers and students to collaborate with others?
2. How do our MURSD collaboration practices compare to other districts with 1:1 programs?
3. How might we capture and measure our teachers’ and our students’ growth in their ability to
collaborate using devices?
4. What next steps are recommended to utilize the devices to enhance collaboration for
student learning?
Core Programmatic Questions

8. Innovate:
1. To what extent do teacher and student uses of the iPad reflect innovation as compared to
Modification and Redefinition stages of technology integration?
2. How do our MURSD practices compare to other districts with 1:1 programs?
3. How might we capture and measure our teachers’ and our students’ growth in their ability to
innovate using devices ?
4. What next steps are recommended to utilize the devices to enhance collaboration for student
learning?
Other:
1. To what extent does the MURSD use of technology reflect student-centered teaching practices?
2. To what extend does the MURSD use of technology reflect personalized learning opportunities?
3. To what extent does the 1:1 program evaluation impact other aspect of student learning outcomes
including, but not limited to, MCAS and academic achievement scores?
4. What other core values or opportunities might MURSD consider when planning the future of
technology integration and evolution in its schools?
Core Programmatic Questions

9. • Three proposals received
• Proposal teams have worked nationally on 1:1 program implementations and evaluations
• Applicants included
• the author of the U.S. Department of Education 2016 Tech plan
• the developer of the SAMR Instructional Model
• the program evaluator for the Harvard / MIT edX Massive Open Online Courses (MOOCs)
• a former state-level ISTE Board President
• Selection Factors
• Multiple on-site visits over an extended period of time
• Clear understanding of our Communicate, Collaborate, Innovate goals
• Ability to provide concrete recommendations to advance teaching practices, particularly in
regards to SAMR
Selection Process

10. Dr. Ruben Puentedura Dr. Steve Stemler
MURSD Evaluation Team
Dr. Damian Bebell

11. • 5 month study of Grade 5-12
• Multiple Data Sources
• Classroom observations
• Teacher & Student Surveys
• Teacher, Student & Parent Focus Groups
• SAMR model used to provide descriptors of the type of learning
experience for student
• Regular contact between MURSD team & research team to navigate
logistics
• Final report presented in September of 2017
The Process

12. ClassObservations
StudentSurvey
TeacherSurvey
StudentFocusGroup
TeacherFocusGroup
LeadershipInterview
How are iPads being used by teachers and students to communicate with others?
How do our MURSD practices compare to other districts with 1:1 programs?
How might we capture and measure our teachers’ and our students’ growth in their ability to
communicate using devices?
What next steps are recommended to utilize the devices to enhance communication for student
learning?
How are iPads being used by teachers and students to collaborate with others?
How do our MURSD collaboration practices compare to other districts with 1:1 programs?
How might we capture and measure our teachers’ and our students’ growth in their ability to collaborate
using devices?
What next steps are recommended to utilize the devices to enhance collaboration for student learning?
To what extent do teacher and student uses of the iPad reflect innovation as compared to Modification
and Redefinition stages of technology integration?
How do our MURSD practices compare to other districts with 1:1 programs?
How might we capture and measure our teachers’ and our students’ growth in their ability to innovate
using devices ?
What next steps are recommended to utilize the devices to enhance collaboration for student learning?
To what extent does the MURSD use of technology reflect student-centered teaching practices?
To what extend does the MURSD use of technology reflect personalized learning opportunities?
To what extent does the 1:1 program evaluation impact other aspect of student learning outcomes
including, but not limited to, MCAS and academic achievement scores?
What other core values or opportunities might MURSD consider when planning the future of technology
integration and evolution in its schools?
CommunicateCollaborateInnovateOther
KEY:
direct data source
indirect data source
Study Instruments Aligned Our
Questions

13. Classroom Observations
146 observations resulted in 3,510 minutes (58.5 hours) of classroom
observations across all major subject areas in grades 5-12:
3 components of MURSD observation rubric:
1. Running Narrative
2. Observation Summary Checklist
3. Time Interval Observation Checklist
Figure 2: Total classroom observations (# of minutes) conducted across grade level

14. Student and Teacher Surveys
Surveys were designed to document and quantify: the variety and extent of students’ and
teachers’ technology use, students’ and teachers’ attitude toward technology, teaching,
and learning, as well as students’ and teachers’ beliefs on student motivation,
engagement, communication and collaboration
Total
student
enrollemt
#of
completed
surveys
#ofIRB
eligible
students
#ofcompleted
IRBelgible
surveys
Response
Rate
MiscoeHill 785 706 562 518 66%
NipmucHigh 619 509 345 309 50%
Total 1404 1215 907 827 59%
Student Survey n=827Teacher Survey n=102
Total
eligible
teachers
# of
completed
and eligible
surveys
Response
Rate
Miscoe Hill 56 53 95%
Nipmuc High 51 49 96%
Total 107 102 95%

15. Focus Groups
Compliment and triangulate survey results and classroom observations
Provide more in-depth information on students’, teachers’ and parents’
experiences and perspectives
Document examples and perspectives towards new opportunities with
communication, collaboration, and innovation
Provide community members an opportunity to share experience, voice,
and perspective
StudentsTeacher Parents
n=36 n=31 n=10
All focus groups were 100% voluntary,
catered, and conducted in May & June 2017

16. Summary of Major Findings (Triangulated)
• Robust amount of student and teacher technology
use (comparable to international 1:1 study results)
• More frequent use and slightly “higher SAMR” use
@ in upper grade levels (despite shorter
implementation period)
• Large degree of positive attitudes and beliefs
towards technology in school
• Limited evidence of technology use beyond
“Substitution” level
• Numerous and varied examples of Innovation,
Communication, & Collaboration

17. General Recommendations and Next Steps
Review and consider different perspective and patterns in full report
Use newly developed tools to survey students and teachers annually
Define and track success indicators against educational technology indicators
Self-evaluate the appropriateness of tablet devices for all learners
Enrich pedagogy with more Augmentation, Modification, and Redefinition
PD models with a more curricular and pedagogical focus (particularly in lower grades)
Provide more technology rich offerings to a wider array of students

18. Key Takeaways
• Be very transparent and explicit about the goals of your 1:1 learning
program
• Involve all stakeholders… and lay the political groundwork where you
have to
• Tying 1:1 learning to traditional metrics (e.g., MCAS, SATs, etc.) is a
fool’s errand
• Have a framework such as SAMR to use as a lens of your practices
• Use program evaluation as leverage to improve instruction!

19. • Find the number in the right hand corner of your index card
• Move to the location in the room where the sign number matches your
card number
• Visit www.tinyurl.com/MURSDMassCUE to access your team’s Google
Doc for this session
• Take 5 minutes to:
• Identify your top priority core value
• Define what it would look like in practice in your classroom
• Draft questions to help you assess the impact that their 1:1
computing program has on that core value
• BONUS: Can you identify a possible data sources to answer this
question?
• Prepare to share questions with your peers
Your Turn

20. Conclusions
• Determine your core values and guiding questions before launching your program
evaluations
• Multiple data sources (surveys, observations & focus groups) are critical to a
comprehensive evaluation
• The benefits of 1:1 Learning opportunities are not necessarily captured by
standardized test scores
• SAMR is about the thinking with tech, not the tech itself
• The evaluation is an ongoing process, not a singular event

21. Questions/Reflections
dquinn@mursd.org
Thank you!