The document describes challenges in analyzing long-term collaborative processes that use multiple tools over time. It discusses both variable-level and event-level approaches, and their limitations. A combined approach is needed to understand how interactions, activities and learning unfold over the duration of the collaboration.

1. "Exploring methodological approaches to
conduct process analysis
in fragmented, long-duration
collaboration mediated by multiple
artifacts."
Jari Laru
Learning & Educational Technology Research Unit
University of Oulu
10/2009

2. Pedagogical design..
Course wiki & Blog (announcements; schedules; lecture materials; workspaces; helpdesk)
Media-sharing services
Theoretical
grounding Blog
Wiki
Individual virtual work
Media Reflection,
Constructing
Collaborative
Lecture Grounding session Meaning-making session virtual work
Media Reflection,
Constructing
Theoretical Groups’
grounding grounding Elaborating Constructing
Media Reflection,
Constructing
Media Reflection,
Constructing
Syndication service via RSS
Monitoring
= Face-to-Face sessions

3. led to messy data..
RSS Feeds RSS Feeds
Media Media
Media
Reflection,
Theoretical Theoretical Media Media Constructing
grounding grounding Media
pre-test Media Media
(conceptual Media Reflection, SRI
Theoretical Theoretical Elaborating Constructing post test
knowledge) Constructing interviews
grounding grounding Media Media
+
questionnaire Media
Media Media Reflection,
Theoretical Theoretical Constructing
grounding grounding Media
time
Event level
media video
lecture video media Blog content video Wiki history
artefacts recordings
topics recordings artefactsmonitored recordings
content
Variable level
statistical Conversation Content Content Content Content Conversation Content statistical conversation
methods analysis analysis analysis analysis analysis analysis analysis methods analysis
Ouput: a lot of different kind of frequency data a la coding categories, log data, conceptual test etc
statistical methods used to calculate
dependencies or probabilities between variables

4. Research problem was
and is uh..challenging
What they did really do with the social
software and mobile devices? A
temporal analysis of long-term
technology mediated collaboration.

5. Methodological skills oriented
my interests towards variable
level approach
• Content Analysis
• Social Network Analysis
• Conversation Analysis Variable level
• Mann-Whitney U-test
• etc..

6. Well, what’s the problem
with variable data?

7. Variable level
We werent’ only using
MULTIPLE tools..
Blog Wiki
but we also had different kind of F2F
settings in different phases

8. Actually results lived in data
archipelago, were small (or
big)islands there

9. Having a holistic approach into
analysis and design, it was little
bit too tricky to start..

10. Event level
Ishikawa diagrams
(also called fishbone diagrams or cause-and-effect diagrams)
FA IL
First event-level try-out. No success. No
methodological bacground. FAIL

11. Event level
A Framework for Analyzing Interactional Processes in
Online Learning
Daniel Suthers, Nathan Dwyer, Richard Medina, Ravi Vatrapu (2007). Paper presented at the Annual Meeting of
the American Educational Research Association (AERA), Chicago, April
9-13, 2007.
FA IL
Too instrumental & multi-widget-chat tool
orientation (several presentations at CSCL’09)

12. What’s next?
• Didn’t found appropriate methodological
Event level literature of event level analysis
• wrong terminology used (didn’t know)
WEB2.0 / Social WEB
• this terminology didn’t help neither
• Methodologically exteremely heterogenous
area

13. Variable level approach
was chosen..
Appropriate coding categories?
Blog Wiki
FA IL K
O

14. Cress & Kimmerle (2008)
A systemic and cognitive view on collaborative knowledge building with wikis.
International Journal of Computer-Supported Collaborative Learning, 3(2), 105–122.
Basic categories:
“Besides assimilating information the wiki can
also accommodate. This happens when new
information is not only attached to the
existing information, but the information in the
wiki is organized in a new way.”
O K

15. Problem remains
Blog Wiki Other
L L
FAI FAI
Data Data Data
Scheme Scheme Scheme
Variables Variables Variables

16. Analytical toolkit for
time-series data
MISSING
Wiki: timetags are essential. No way to import into
NVIVO8
Blogs: cumulative nature and “metadata” essential. No
way to import into NVIVO8
WHERE IS A TOOL FOR ANALYSING
HETEROGENOUS, MULTITOOL TIME-SERIES
DATA? Answer: Tatiana! (CSCL 09)

17. TIME IS RUNNING!
New tryout with variable
level approach
• How to measure dependencies
between (all) variables??!
• Petri Nokelainen visited our
group: IDEA! Bling!

18. Bayesian dependency modeling
• SRL Self-report questionnaire (Pintrich, 1993; Wolters, 2001)
• Short-answer (paper & pencil) conceptual knowledge test
– 3 x 6 concepts (about lectures), same in pre/post tests
• Basic activity measures as “log data”
– # of images/videos (file-sharing services)
– # of articles/edits (blog, wiki)
– # of wordcount (blog, wiki, face-to-face sessions)
– # of items (rss)
• Qualitative data collection for other analyses (stimulated recall interviews, group interviews, content produced by learners, wiki
history)
• Statistical dependencies between the variables were studied with
Bayesian Dependency Modeling (BDM), which predicts the most
probable statistical dependency structure between the observed
variables (Myllymäki, Silander, Tirri & Uronen, 2002).
• B-Course: A Web-Based Tool for Bayesian and Causal Data Analysis
was used as a analysis toolkit
• Bayesian analysis is based on propabilities instead of frequencies
(Nokelainen, Silander, Ruohotie, Tirri, 2007)
var 1 var 2
B-Course

19. 19
Are learners’ actions in the web2.0 tools and
F2F sessions related to their learning outcomes?
wiki
Learning outcome
“monitor”
Syndication service direct dependencies:
Learning gain:
- RSS_Read (# of read RSS items)
FA IL => Monitor
- WP_Entries (# of blog entries)
=> Reflector
wiki
mediasharing
“reflector”
Some arcs are pretty strong, so that removing any of them
would result in a model with probability less:
* than one millionth of that of the original model
blog
** than one thousandth of that of the original model
LET - Learning and Educational Technology Research Unit *** smaller than one thousandth (weak connection)
Jari Laru. 27.8.09 ERLI09, Amsterdam

20. 20
WHY IT WAS DOOMED TO BE A
FAIL?
• Bayesian approach provided only surface
level analysis:
–deep insight into collaborative interaction
in F2F groups, wikis & blogs missing
–order, duration and quality of events: eg.
lecture > F2F > image/video > blog reflection > F2F > Wiki
• Interesting question “How learning unfolds
over time” wasn’t answered at all..
• Didn’t support design of the study!
LET - Learning and Educational Technology Research Unit
Jari Laru. 27.8.09 ERLI09, Amsterdam

21. OOPS!

22. What’s next?

23. What I had on my
hands after Bayesian?
• Analytical toolkit: Tatiana
• Some coding schemes which didn’t stretch
over the whole design
• Some methodological hints about event
based approach..
• Intuition that event based approach is “my
way”

24. Task level
Narrative and learning with web2.0
technologies: towards a research agenda.
Pachler & Daly (2009). Journal of Computer Assisted Learning. 25, 6-18
•Narrative trails (+)
•Sense-making (+)
•Blog centric (-)
•vague understanding
of collaboration (-)
(commenting in blogs)
•Holistic approach?..
m m
e ll..h
W

25. Missing piece is ..
Missing piece

26. Time is precious:Variable- and event-centred
approaches to process analysis in CSCL research.
(2009) Computer-Supported Collaborative Learning (2009). 4:239-257
Methodological link
Mercer (2008) et. al Muukkonen et. al
Suther et. al (2007)
communication & (2008)
graphical techniques
learning in classroom Time series analysis
Central tenet: “methods developed in conversation analysis and
ethnomethodology...do not carry over to interactions that are
fragmented over time, stretch over longer durations, or are mediated
by artifacts rather than talk” pp.241

27. Process method should..
(Reimann, 2009)
• a process method should, in addition to
efficient cause (typical for event approach),
be able to deal with at least two other
kinds of causes:
• formal cause, referring to the patterns of
which things are made
• final cause, the end for which things are
made [goal, in our study: wiki]

28. One coding scheme to evaluate
interpersonal exchange
• Mercer, N. (2008). The seeds of Time: Why Classroom
Dialogue Needs a Temporal Analysis. The Journal of
Learning Sciences, 17: 33-59
• Focus of article is on examining long-term classroom
talk and how it’s used to:
• represent past experiences
• carry ideas forward from one occasion to another
• approach future activities
• achieve learning outcomes

29. I propose an applied
approach for Mercer
• To evaluate interactions that are mediated by
artifacts or than talk and can be categorized
according Mercer:
• represent past experiences [for example: in personal
blog, or F2F discussions]
• carry ideas forward from one occasion to another
[photo => blog, or peer blog => own blog etc.]
• approach future activities [design groups’ / individual
activities]
• achieve learning outcomes [troughout the design]

30. Unit of analysis in my
research
• Main entities are individuals and groups
• events are constrained to those incidents
in which either individuals or groups can
participate
• Focus will be put on the sequences of
activities, incidents, crises, or stages that
unfold in the groups over the duration of
the course

31. led to messy data..
RSS Feeds RSS Feeds
Media Media
Media
Reflection,
Theoretical Theoretical Media Media Constructing
grounding grounding Media
pre-test Media Media
(conceptual Media Reflection, SRI
Theoretical Theoretical Elaborating Constructing post test
knowledge) Constructing interviews
grounding grounding Media Media
+
questionnaire Media
Media Media Reflection,
Theoretical Theoretical Constructing
grounding grounding Media
time
Event level
media video
lecture video media Blog content video Wiki history
artefacts recordings
topics recordings artefactsmonitored recordings
content
Variable level
statistical Conversation Content Content Content Content Conversation Content statistical conversation
methods analysis analysis analysis analysis analysis analysis analysis methods analysis
Ouput: a lot of different kind of frequency data a la coding categories, log data, conceptual test etc
statistical methods used to calculate
dependencies or probabilities between variables