How to Add a New Field in Existing Kanban View in Odoo 17
Using Learning Analytics Data to Understand Project-Based Learning
1. Using Learning Analytics
Data to Understand
Project-Based Learning
The Case of the Carrot
Wizards in Globaloria-
West Virginia
ICLS 2012 Pre-Conference Data Workshop
***
Rebecca Reynolds, Ph.D.
Assistant Professor, Library & Information Science
Rutgers University School of Communication and
Information
2. Background Info
• The following slides provide background on
the Constructionist / design-based research
innovation, “Globaloria,” being implemented
by a NYC non-profit called the World Wide
Wokrshop, in several US states with grant and
state education department support (WV, TX,
CA, FL, NY)
3. Globaloria Game Design Program Learning Objectives: Cultivate the Six
Contemporary Learning Abilities (6CLAs)
Developing games in a social learning system offers students
Constructionist, participatory design and programming experiences
leading to dispositions, practices and expertise necessary for successful
engagement in today’s digital, knowledge-based economy
4. The Six Contemporary Learning Abilities (6CLAs):
Developing games in a social learning system offers participatory
experiences leading to practices necessary for engagement in today’s
knowledge-based economy
6. Globaloria-California
California, 2011/2012:
• Supported by $1M Knight
Foundation grant
• “Formal” implementation
at 2 middle schools (67
students)
• “Informal” implementation
at 2 Boys and Girls Club
community centers (38
students)
• First year of California
participation
• Potential to compare
implementation context
variable differences (time
on task; provision of grades
vs. none; linkage to core
curriculum vs. open choice
game topics, etc.)
7. Globaloria-TX
East Austin College
Prep, 2011/2012:
• 1 charter middle
school
• 3 grades (6th, 7th,
8th)
• ~250 students
• 3rd year of
school’s
participation
• 2 educators
• Unique
longitudinal
implementation
(all EACPA
students take
Globaloria daily,
for credit and a
grade for 3 years)
8. Globaloria-West Virginia2011/2012:
• 5th year of Globaloria
implementation in West Virginia
• Individual schools and teachers
range in N of participation years
• Majority of students, 1 year only
• 17 middle schools
• 50 high schools
• All in-school implementation,
daily, for credit, for grade
• Mix of elective game design
classes, and required classes
linked to the core curriculum
• Multi-level analysis and
implementation context variable
differences increasingly being
measured (e.g., teacher N of
participation years; extent of
teacher training; student game
subject focus;
9. Domains of Learning and Expertise
• Constructionist digital literacy (skills needed in knowledge
economy => 6-CLAs)
• Computational thinking through game design in Flash and
programming in Actionscript
• Core curricular subject matter:
– When game subjects are linked to core curriculum and students
deepen knowledge about topic through online research and
design
• STEM career interests: Technology & Engineering;
Computer Science
• Motivation, Affect, Attitudes, Life Choices, New Possibilities
and Horizons, esp. for disadvantaged students
10. Flash software, Wiki Environment, Curriculum,
Tutorials
“Hands On” Training Sessions (virtual, local)
• Globaloria Academy – In-person, intensive trainings (3)
• Online Mini Webinars - Web-based workshops (7)
Globaloria Mentors Program
Experienced educators take on a leadership role by
supporting other educators
“24/7” Virtual Support
• Expert Support via wikis, blogs, email, WebEx
• Educator Community Development – private
educators community wiki, peer-to-peer mentoring, weekly
educators newsletter, sharing teaching & learning reports
Rewards and Recognition
•Teachers: Stipends and Graduate credits are earned
•Students: Nationally-Recognized Game Design
Competitions
Learning Supports for Students and Educators:
12. Syllabus integrates learning about the
game subject topic and
game programming and mechanics
Syllabus
Topics
Addressing
Game Subject
Syllabus Topics
Addressing Game
Programming and
Mechanics
www.myglife.org/usa/wv
19. Some Globaloria project research
goals at a high level
• Understanding the influences and forces at play in this social learning
system; mapping the terrain from a social science perspective
• Design-based research: Iterative improvement of the social learning
system; generalization towards learning theory, and the instructional
design of effective learning innovations
Connecting multiple levels of analysis in our research will be necessary to accomplish
the above . . . And ideally, the below:
• Evaluative Diagnostics:
– Use predictive modeling to identify trajectories of student success / failure
across time (what factors at multiple levels of analysis influence student
learning?)
– Apply predictive findings towards development of diagnostic tools for early
intervention that teachers can use for formative assessment
• Summative Evaluation: Developing tools for program evaluation and
student assessment
20. 2 Main Strands of Research
Quantitative multi-level analysis modeling (w. Ming Ming Chiu)
• Purpose: Model the complex relationships among variables of importance. Currently,
DV is evaluation scores for games based on content analysis of their quality
• Present data sources:
– Teacher and student pre / post surveys
– Teacher progress reports
– Content analysis evaluation of team games
– Wiki log files and behavioral site metrics data (edits, uploads)
– Other data TBA (e.g., game subjects; teacher grades; student role-taking; resource use)
Qualitative analysis, comparison of cases (w. Cindy Hmelo-Silver)
• Purpose: investigate and explain dynamics / inter-relationships among variables;
identify key phenomena driving the system; add to quantitative model as relevant
• Present data sources:
– Semi-structured teacher and student team interviews at two timepoints this past semester
(18 middle school teams, TX & WV, and their educators)
• Main themes: Teacher/peer/e-learning/non-profit scaffolding; student problem self-structuring;
student/teacher resource use; areas of challenge/enjoyment; teamwork processes
– Future: Participant observation in situ in the classroom
21. Challenging Assumptions
• To what extent do complex multi-level phenomena and
dynamics in this system break down into discrete
variables that can be investigated for their inter-
relationships in quantitative modeling? Is the
phenomena in such a social learning system better
described quantitatively? Qualitatively? Mixed
methods? How can varying analyses inform each
other?
– Perhaps something we can discuss today
23. Mean pre-intrinsic
motivation
+0.176 *
Mean increase in
intrinsic motivation
+0.163 *
Mean SWF flash
uploads per month
+0.166 *
Mean age –0.276 ***
Team Structure
Design mixed
genre game
+0.833 **
Team Processes
Design
Game play
Concept
Mean self-
learning hours
Programming
–0.452 ***
Teacher Outcome
24. Analysis highlights
3 levels
Classroom (teacher)
Team (mean of group members)
Team process (mean of team processes)
Test hypotheses about outcomes
w/ explanatory variables at lower levels
--characteristics of samples
Mean, SD, max, min, …
Model multiple outcomes simultaneously
25. Qualitative analysis
1. What are the theoretical instructional design assumptions implicit in the
design and development of Globaloria?
– Guided discovery-based learning, building on Constructionism
– Scaffolding provided via a range of structures (explored in research)
2. How are students at different schools interacting with the range of
scaffolds provided? Autonomous/controlled/task-driven
– A focus on student self-structuring of problems and resource uses to support their
learning, at various stages of the process
3. Linking Scaffolds, Problem-Solving, and Resource Use:
– To what extent does it appear that the structuring elements / scaffolds above help
students meet learning goals: conceptualize the idea for a game, develop a prototype,
implement a game in Flash, learn programming in Actionscript, complete the final game,
do this in a team context (combining files), and upload the final game to the wiki?
4. Does this process appear to bring about other outcomes besides discrete
CLAs/Constructionist digital literacy, and/or other domains of expertise
(computational thinking; STEM career interest)
– Resourcefulness?
26. Approach to the ICLS 2012
Data Workshop
What can we learn about multi-level phenomena from
an in-depth focus on 1 class of 2010/2011 game
designers, using an eclectic array of data sources?
• Investigate the experience of a single case study team of
game designers from TVHS, consisting of 3 HS students who
won the annual game design competition in the 2010/2011
school year with their Civics content game, “Henry the
Hedgehog”
– Understanding team processes can expose some of the
values/norms built into the system (what makes a good game?)
• Investigate relationships among variables that appear
influential to learning.
• Consider this case in context, given that similar data are
available for over 2000 students across 3+ years
• Apply findings to ongoing research being conducted
27. Data Sources Provided to Participants, and Corresponding Levels of Analysis
Level of
Analysis
Local context Data source Filenames
Innovation Curriculum Description of Globaloria syllabus,program
overview
2.Curriculum.pdf
ICLS2012_dataworkshop.pptx
E-learning
platform
Description of features of e-learning platform;
screenshots; wiki log files (trace data of student
activity on the wiki)
2.Curriculum.pdf
ICLS2012_dataworkshop.pptx
13. Aggregate wiki log _ contentanalysis.xlsx
System WV
Department
of Education
Game competition press release; information on
judges and judging criteria.
14.game_comps_press_2011-1.pdf
SampleWikiCurricLesson_Drag and Drop.docx
WikiScreenshots folder
Organizational School Description of the school 3.Case Study Data File, Carrot Wizards.docx
ICLS2012_dataworkshop.pptx
World Wide
Workshop
non-profit
Description of non-profit, goals 3.Case Study Data File, Carrot Wizards.docx
ICLS2012_dataworkshop.pptx
Class Teacher Progress report from teacher offering reflection on
PD experiences, learning and teaching; and
observations of students -- individuals and teams
4.teacherprogress report.doc
Team Game design
teams
Team key (who is in what team); Carrot Wizards
case study report; evaluation of team games; wiki
log files
5.teammembers_fullclass_tvhs_y4.csv
7.1.CarrotWizardsFINALGAME.docx
8.yr4_globaloria_eval_researchfile_TVHS.xlsx
13. Aggregate wiki log _ contentanalysis.xlsx
Individual Student
participants
Pre- and Post-Surveys; student blog URLs; case
study report; wiki log files
9.yr4_globaloria_postsurvey_qualitative_TVHS.xls
10.yr4_globaloria_presurvey_qualitative_TVHS.xls
11.blogURLsforCarrotWizards.docx
13. Aggregate wiki log _ contentanalysis.xlsx
Artifact Student
games;
student wiki
work
Actual team games; wiki screenshots; game
evaluation
6.File_Cwfinal2atyler.fla
7.1.CarrotWizardsFINALGAME.docx
7.File_Cwfinal2atyler.swf
3.Case Study Data File, Carrot Wizards.docx
WikiScreenshots folder
8.yr4_globaloria_eval_researchfile_TVHS.xlsx
28. Further context about TVHS
• WV high school class at TVHS took Globaloria as a game design
elective, meeting daily for 90 minutes per day, for credit and a
grade across an entire school year (2010/2011)
• This was TVHS and educator A. Leef’s first year in the program
implementing Globaloria; she participated in the standard
Globaloria Academy PD training and ongoing offerings throughout
the year.
• Students in 8 teams of 1, 2 or 3, created a total of 8 games about a
range of topics relating to either Civics or STEM
• 2 games focused on Civics: Carrot Wizards’ Henry the Hedgehog
game, and Skull Crusher’s game Blast.
• Henry the Hedgehog game won the Globaloria Annual Civics Game
Design competition in the 2010/2011 school year
29. Analysis: Narrowing in on Team Process step of
quantitative multi-level analysis, using Wiki log files
• Behavioral trace log file data available, measuring
individual-level edits to wiki pages, and uploads of
game artifacts, graphics, .FLAs and .SWFs.
• Research Questions:
– How do edits / uploads of teams at TVHS vary across time?
• Totals, team means, indivs-within-teams
– General observations about wiki use patterns
– What assumptions/limitations associated with using
frequency data in multi-level analysis?
– What stories do the data tell, that the aggregate frequency
numbers miss?
30. Preliminary Researcher Analysis of
Data Sources: Wiki log files
• Edits to team pages (main locus on the wiki for team-level activity):
• Game project file uploads
31. Team Rankings,
Total Sum Evaluation Score
Team Name:TOTAL
SUM
Carrot_Wizards 57
Shock-a-holics 53
LEET_HAX 51
Bull_Loany_Productions 50
Gummy worms.(: 49
Thomas the Train's
Pancakes of Pain 49
TEAM_SKULL_CRUSHER 46
TOMORROW 19
32. Aggregate Sum Results, Edits
Team Name
N of Team
Members Total Edits Mean Edits
Carrot_Wizards 3 161.00 53.67
Gummy_worms.(: 2 144.00 72.00
LEET_HAX 1 66 66
Shock-a-holics 3 91 30.33
TEAM_SKULL_CRU
SHER 2 60 30
Thomas_the_Train
's_Pancakes_of_Pa
in 3 60 20
TOMORROW 1 12 12
33. Descriptive Results
Total Edits to Wiki Team Page
Week of 9/20/10 – week of 5/9/11
0.00
20.00
40.00
60.00
80.00
100.00
120.00
140.00
160.00
180.00
Total edits
Mean edits
34. Total N of Edits to Team Wiki Pages,
Week of 9/20/10 – week of 5/9/11
How does team activity vary across time?
• Carrot Wizards appears to have invested greater effort in editing their team wiki page at
the beginning than other teams. Evidence of possible impact of early planning.
• Carrot Wizards and Gummy Worms appear most consistent in their wiki editing across
time
• Carrot Wizards appears to have exerted more effort towards the end of the school year
than other teams
Note: 1 outlier data point from a single day by a single student in 2010w12, Leethax team omitted
35. Mean Edits to Team Wiki Pages,
Week of 9/20/10 – week of 5/9/11
(Mean = Total N of page edits / N of team members)
How does standardizing for N of team members influence patterns?
• Gummy worms had fewer team members, thus logged more average activity with consistency as a
team
Note: 1 outlier data point froma single day by a single student in 2010w12, Leethax team omitted
36. Explaining Gummy Worms greater N of
edits (individuals within teams)
• Consistency in editing among both team members
37. Explaining Gummy Worms greater N of
edits (individuals within teams)
• Tyler was primary wiki team page editor. The other 2 team members
only did so in S2 at the end.
• What is the difference in team dynamic, between having 2 vs. 3 team
members?
38. Total N of File Uploads,
Week of 9/20/10 – week of 5/9/11
How does team activity vary across time?
• All teams have greater frequency/consistency of uploads in S1; uploads become more
concentrated during S2
• File types vary to a greater extent in S1 (includes more static graphics files)
• Carrot Wizards and Gummy Worms files are mainly SWFs and FLAs in S2
39. Explaining Gummy Worms greater N of
uploads (individuals within teams)
• Brianna uploads most frequently and consistently
• Nicole outlier in week 30
40. Carrot Wizards individuals-within-team
Uploads
• Relative consistency among team members
• What was occurring at week 29-34 in the program curriculum to spark high uploads
among these 2 teams?
41. Connecting team and individual levels
of analysis
• How is wiki work distributed across individual
team members?
– Procedure:
• For Carrot Wizards only, reviewed log files across
time, by team member
• Review actual wiki content edits made by team
members across time
• Note categorical types of wiki content edits
• Make observations regarding individual activity within
the team
– Link inferences to team-level findings above
47. Connecting Wiki activity
to Evaluation Scores
• To what extent do patterns of wiki activity
relate to evaluation outcomes?
– This is a next step in our analysis.
– We welcome the input of other researchers
48. Next Steps
• Summarize findings on individual
contributions within the team context
• Consider results in relation to game evaluation
findings
49. Opportunities for Workshop
Participants
• Conduct social network analysis / sequential
interaction analysis on individual and team level
wiki activity results for TVHS teams
– Raw aggregated wiki log file and evaluation outcome
data are available in the Excel file, “13. Aggregate wiki
log _ contentanalysis.xls”
• Relate wiki raw frequency and social network
analysis findings to final game evaluation
outcomes
• Explore relationships between qualitative findings
in Dropbox, and results reported herein
Notas del editor
How do we plan to grow exponentially in a way that is sustainable? Strong and innovative Professional Development programs for educators, principals and students ensure the community can self-manage, grow and develop.Hands-on training: Mentor program – cascading and taken to scaleVirtual support – sustainable, scalable – walking our talkRewards & Recognition – we pay
In addition to the amazing impact we have seen over the past few years researching and evaluating the Globaloria program we…We are in unique position of having everything we do be rooted in years of academic research and real-world practice. In the 1980s and 1990s, while at Harvard and the MIT Media Lab, Idit really founded the concept that children learn best by designing – when they are programming computers instead of computers programming them. Ground breaking research with children in Bronx, showing how software designing could change their relationship to education and their engage them in learning in a way that had not been seen before
In addition to the amazing impact we have seen over the past few years researching and evaluating the Globaloria program we…We are in unique position of having everything we do be rooted in years of academic research and real-world practice. In the 1980s and 1990s, while at Harvard and the MIT Media Lab, Idit really founded the concept that children learn best by designing – when they are programming computers instead of computers programming them. Ground breaking research with children in Bronx, showing how software designing could change their relationship to education and their engage them in learning in a way that had not been seen before
In addition to the amazing impact we have seen over the past few years researching and evaluating the Globaloria program we…We are in unique position of having everything we do be rooted in years of academic research and real-world practice. In the 1980s and 1990s, while at Harvard and the MIT Media Lab, Idit really founded the concept that children learn best by designing – when they are programming computers instead of computers programming them. Ground breaking research with children in Bronx, showing how software designing could change their relationship to education and their engage them in learning in a way that had not been seen before
We have taken all this learning and experience and established the cornerstone of the Globaloria Program – It’s Learning Formula. I want to take a minute on this slide: Self-Led Learning- students determine the needs of their learning as they progress through the process of designing their own games. It is purposeful, self-directed and hands-on learingPeer-to-peer learning – it is collaborative learning. Students take on the role of both educators and students. They have a critical role in their own learning and in the advancement of their learning community. They are invested. Co-Learning – we are breaking down the traditional hierarchy in the classroom and having students and educators learn together. Students don’t learn for the grade, but as part of a community of creators. Expert-guided learning – we do not divorce learning from the real-world…it is the real world. Students engage with real professional and see the relevance of their learning in today’s world.