Machine Learning Model Validation (Aijun Zhang 2024).pdf
Are Great Wikis Born or Made? Are Students Just Posting in the Same Place?
1. Are Great Wikis Born or Made?
Evaluating Quality Trajectories in
Wiki Learning Environments
Justin Reich
Richard Murnane
John Willett
Just Posting in the Same Place?
Evaluating Student Collaboration in
Wiki Learning Environments
@bjfr
edtechresearcher.org
2. Distributed Collaborative Learning
Communities Project:
Web 2.0 in K-12 Settings
• Excellence: How do we make them
good?
• Equity: Do only certain kids get the
good ones?
• Analytics: What can we learn about
learning from real-time usage data from
online learning environments
2
5. Low Income
Schools
(n=117)
Mid to High
Income Schools
(n=133)
Failed or Trial Wiki 50% 30%
Teacher-Content Wiki 34% 35%
Individual Student-Owned
Wiki
15% 35%
Collaborative Student-Owned
Wiki
2% 1%
Median Lifetime 6 days 33 days
7. “Time is Precious”
• Reimann (2009)
– Collaborative learning unfolds over time
• Substantive need
– Online learning environments track data in
real time
• Methodological opportunity
9. Theoretical debates in wikis and collaboration (IJCSCL)
Are students meaningfully
collaborating?
Dohn (2009): Web 2.0 tools
are theoretically well-suited
for collaborative knowledge
production practices, but the
educational logics of
individual assessment
prevent these practices from
being realized.
Or just posting in the
same place?
Glassman and Kang (2011):
wikis not only enable
students to participate in
knowledge building, but
allow the enactment of
Dewey’s vision
10. The Plan
• Explain our research design
• Explain how we model wiki quality over
time
– Answer the question: Are Great Wikis Born or
Made?
• Explain in detail how we measure
collaboration
– Answer the question: Who’s right: Dohn or
Glassman & Kang?
12. Which wikis are in my dataset?
• All 179,851 publicly-viewable education-related
wikis started on the PBworks platform between
June 2005 and August of 2008.
• Does not include “private” wikis (~70,000)
12
179,851
PBWorks Wikis
13. Which wikis are in my sample?
• Randomly sampled 1,799 wikis (1%)
• Coded to identify 406 U.S. based, K-12 wikis
• 255 from specific, identifiable public schools
– Detailed usage statistics provided by PBworks.com
– School level SES data from the Common Core of Data (National
Center for Education Statistics, 2007-2008)
– 255 wikis are from 41 U.S. states
13
179,851
PBWorks Wikis 1,799
1%
Random
Sample
406
U.S. K-12
Wikis
255
Public
School
15. Wiki quality as opportunities for
21st Century Skill Development
Expert
Thinking
Complex
Communi-
cation
New Media
Literacy
16. Wiki Quality Instrument
• Information Consumption (2 items):
– Do students use wikis to get information? links?
• Participation (4 items):
– Do they contribute? Do they own pages?
• Expert thinking (5 items):
– Do students use academic content knowledge in wiki activities?
– Do students reflect on the process/product?
• Complex Communication/Collaboration (7 items):
– Do students concatenate text on pages?
– Do they substantively edit each others work and co-create pages?
• New Media Literacy (6 items):
– Do students use formatting?
– Do they hyperlink? Embed media?
24 Questions
Wiki Quality Scale
of 0-24
17. How did we measure wiki quality?
• Sample wiki quality at 7, 14, 30, 60, 100, and
400 days
• Two raters independently apply wiki quality
instrument
– All raters must code “training set” of wikis within 1.5
points of master coders
– Weekly meetings while coding to discuss categories,
difficult cases, etc.
• Third rater reconciles disagreements
17
18. Data Analytic Strategy:
Multilevel Model for Change
• We should have time nested within wikis and
use the MMFC to model quality trajectories
• What functional form should we use to
represent time?
• The decision that we make in setting a
functional form will answer the research
question about time!
25. Prototypical wiki quality trajectory (n=406)
0
5
10
15
20
0 100 200 300 400
Composite
Wiki
Quality
Score
Days
26. Implications
• Early wiki quality is a strong predictor of
later wiki quality- Great Wikis are Born
– Hypothesis: Norms developed early on in wiki
learning communities are powerful in shaping
later use
– FOR TEACHERS: Therefore, wiki design
deserves critical attention
– FOR RESEARCHERS: Further qualitative
studies of wiki using communities should
ensure that observers can study wikis from
the start of the process
28. Theoretical debates in wikis and collaboration (IJCSCL)
Are students meaningfully
collaborating?
Dohn (2009): Web 2.0 tools
are theoretically well-suited
for collaborative knowledge
production practices, but the
educational logics of
individual assessment
prevent these practices from
being realized.
Or just posting in the
same place?
Glassman and Kang (2011):
wikis not only enable
students to participate in
knowledge building, but
allow the enactment of
Dewey’s vision
30. Complex Communication
Concatenation Do multiple students add discrete sections of text to the same
page?
Copyediting Does at least one student copyedit text created by another
student?
Co-ConstructionDoes at least one student substantively edit text created by
another student?
Commenting Does at least one student comment upon another student’s
work on the wiki?
Discussion Do students respond to each others’ comments for at least four
conversational turns?
Scheduling Do students schedule meetings or tasks?
Planning Do students plan for future work?
31. Concatenation
Concatenation Do multiple students add discrete sections of text to the same
page?
Copyediting Does at least one student copyedit text created by another
student?
Co-ConstructionDoes at least one student substantively edit text created by
another student?
Commenting Does at least one student comment upon another student’s
work on the wiki?
Discussion Do students respond to each others’ comments for at least four
conversational turns?
Scheduling Do students schedule meetings or tasks?
Planning Do students plan for future work?
32. Complex Communication
Concatenation Do multiple students add discrete sections of text to the same
page?
Copyediting Does at least one student copyedit text created by another
student?
Co-ConstructionDoes at least one student substantively edit text created by
another student?
Commenting Does at least one student comment upon another student’s
work on the wiki?
Discussion Do students respond to each others’ comments for at least four
conversational turns?
Scheduling Do students schedule meetings or tasks?
Planning Do students plan for future work?
33. Co-Construction
Concatenation Do multiple students add discrete sections of text to the same
page?
Copyediting Does at least one student copyedit text created by another
student?
Co-ConstructionDoes at least one student substantively edit text created by
another student?
Commenting Does at least one student comment upon another student’s
work on the wiki?
Discussion Do students respond to each others’ comments for at least four
conversational turns?
Scheduling Do students schedule meetings or tasks?
Planning Do students plan for future work?
34. Complex Communication
Concatenation Do multiple students add discrete sections of text to the same
page?
Copyediting Does at least one student copyedit text created by another
student?
Co-ConstructionDoes at least one student substantively edit text created by
another student?
Commenting Does at least one student comment upon another student’s
work on the wiki?
Discussion Do students respond to each others’ comments for at least four
conversational turns?
Scheduling Do students schedule meetings or tasks?
Planning Do students plan for future work?
35. 0
5
10
15
20
0 50 100 150 200 250 300 350 400
Complex
Communication
Score
Days
Prototypical wiki trajectory for Complex
Communication WQI subdomain (n=406)
40. SourceForge
• 70% of projects have only 1 developer,
• 14% of projects have 2 developers
• 14% of projects have 3-9 developers
• 2% of projects have more than 10 developers
41. Theoretical debates in wikis and collaboration (IJCSCL)
Are students meaningfully
collaborating?
Dohn (2009): Web 2.0 tools
are theoretically well-suited
for collaborative knowledge
production practices, but the
educational logics of
individual assessment
prevent these practices from
being realized.
Or just posting in the
same place?
Glassman and Kang (2011):
wikis not only enable
students to participate in
knowledge building, but
allow the enactment of
Dewey’s vision
42. Implications
• Empirical: Few wikis provide
opportunities for student collaborative
writing
– Very little copyediting or rich discussion
– “I DON’T CARE DO UR OWN PAGE!,”
(Grant, 2009).
43. Implications
• Hopeful: Research to develop strategies
for scaffolding online student collaboration
is really important
• Despairing: What does it mean if Dohn is
right? What if the institutional structures of
schooling limit collaborative learning?
48. What subjects are wikis used for? (n=411)
2
4
5
6
8
10
20
22
26
45
60
61
70
120
0 20 40 60 80 100 120 140
Education
Classics
ESL
Business
Health/PE
Modern FL
Contained Elementary
Art
Library
Math
Computer Science/ Technology
Science
Social Studies
English / Language Arts
49. 49
1,000K
10K
1,000
100
10
1
100K
# of
Cases
Days Weeks Months Years
Design
Research
Interviews
Surveys
Simulations
Semantic
Analysis
Content
Analysis
Discursive
Analysis
Observational
Research
Duration of data collection and capture
Time/Scale Web 2.0 Research
State Space Modeling
Usage
Statistics
Seconds
Biometric
Analysis
50. Prototypical wiki quality trajectories in subject
areas, controlling for SES (n=259)
0
5
10
15
20
25
0 100 200 300 400 500
Wiki
Quality
Score
Days
Social Studies
English
Computer Science
Science
Math
55. AL L PS
0
1 0
2 0
3 0
d a y
0 1 0 0 2 0 0 3 0 0 4 0 0 5 0 0 6 0 0 7 0 0 8 0 0 9 0 0
Moving average of wiki development
measured in page saves
Notas del editor
We took all 179,851 publicly viewable education related wikis created on the Pbworks platform from the start of the company in June 2005 through August of 2008.
We randomly sampled from these wikis to get a 1% sample of 1,799, and then we separated out all of the wikis used in other countries, or in higher education, or that were unidentifiable and we ended up with 411 wikis used in U.S. K-12 settings. We then focused in on a subsample of 255 wikis used in specific, identifiable public schools.
What’s exciting about this random sample is that we have both detail usage statistics from Pbworks, we can example the entire historical content of each wiki, and since these wikis are linked to specific public schools we can obtain school level SES data from the Common Core of Data. We were very happy to see that these 255 wikis come from 41 US states, which provides strong evidence that our random sample can be used to characterize the state of wiki usage across the U.S.
The wiki quality instrument is a coding instrument used by trained researcher, and it consists of 24 questions about different kinds of behaviors typically encountered on wikis in five different categories. REVIEW BRIEFLY.
So you can imagine that our research assts sit down in front of a computer, they pull up a special browser interface that allows them to easily navigate the historical record of wikis, and the look at the revision history and evaluate which of these 24 behaviors are present. The items ask questions LIKE.
So from these 24 dichotomous questions, we can assemble a Composite Wiki Quality Score that ranges from 0 to 24.
So from these empirical analyses and from a comprehensive modeling strategy we ended up selecting a multi-level Poisson regression model as our final model.
And I want to put this model up here to signal methodological competence, to let you know that we did some pretty sophisticated, thorough, careful modeling,
but actually I’m not going to take the time unpack this, and at the end of it all we convinced ourselves that you could very responsibly summarize these wiki quality trajectories not with complicated models by simply reporting the wiki quality score on day 14
Let’s go back to these plots, and pull up one example
Essentially, since wiki quality growth is logarithmic, rapid early on, leveling afterwards, so for most purposes rather than using a complex multilevel statistical machinery to represent the whole wiki quality trajectory, for many purposes you can use just use wiki quality scores at day 14 as a single number, comparable across wikis that persist for different lengths of time, that summarizes wiki quality trajectories. It’s a little bit like using median lifetime as a summary statistic for a survivor function. So we’ve tried to find the simplicity on the far side of complexity.